Metadata deletion method and apparatus, computer device, and storage medium

Abstract

The application relates to a metadata deletion method and device, a computer device and a storage medium. The method comprises the following steps: detecting, by at least one scanning thread, a deletion prefix of each metadata in a to-be-deleted data table; for any scanning thread, if a target deletion prefix corresponding to the scanning thread is detected, the data key value of target metadata corresponding to the target deletion prefix in the to-be-deleted data table is cached in the storage space of the scanning thread; and based on the data key value cached in the scanning thread, a data deletion request is sent to a plurality of deletion threads, so that the deletion threads delete target metadata corresponding to the data key value from the to-be-deleted data table according to the data key value. The method can improve the deletion efficiency of metadata.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for deleting metadata. Background Technology

[0002] Metadata deletion is divided into foreground deletion and background deletion. After a user deletes metadata in the foreground, the metadata is no longer accessible to the user, but it still exists in the system and is actually deleted in the background at a certain rate.

[0003] Existing methods for deleting metadata in the background typically involve setting up two data tables. When metadata is deleted from a table accessible to the front end, the system moves the metadata to another data table in the background. The background then iterates through this data table and deletes the metadata within it.

[0004] However, existing solutions can only perform simple traversal and sequential deletion of metadata in the data table. In terms of deletion rate, they can only control the deletion rate by simply adjusting the number of deletion threads and thread priority. The adjustment method is not flexible enough, resulting in low efficiency of metadata deletion. Summary of the Invention

[0005] Therefore, it is necessary to provide a metadata deletion method, apparatus, computer device, and storage medium to address the aforementioned technical problems.

[0006] Firstly, this application provides a method for deleting metadata. The method includes:

[0007] The deletion prefix of each metadata in the data table to be deleted is detected by at least one scanning thread;

[0008] For any of the scanning threads, if a target deletion prefix corresponding to the scanning thread is detected, the data key value of the target metadata corresponding to the target deletion prefix in the data table to be deleted is cached in the storage space of the scanning thread;

[0009] Based on the data key-values ​​cached in the scanning thread, data deletion requests are sent to multiple deletion threads, so that the deletion threads delete the target metadata corresponding to the data key-values ​​from the data table to be deleted.

[0010] The metadata deletion method provided in this disclosure can distinguish between scanning threads and deletion threads, and use multiple scanning threads to perform concurrent scanning of metadata, thereby improving the scanning speed of metadata and thus improving the efficiency of the scanning threads submitting data deletion requests to the deletion threads. At the same time, by distinguishing between scanning threads and deletion threads, the scanning process and the deletion process can be decoupled, avoiding mutual interference between scanning and deletion that would lead to low efficiency in scanning and deletion, thus improving the efficiency of metadata deletion.

[0011] In one embodiment, before detecting the deletion prefix of each metadata in the data table to be deleted by at least one scanning thread, the method further includes:

[0012] Upon receiving a deletion instruction for the metadata, the deletion prefix of the metadata is generated based on the data identifier of the metadata;

[0013] The metadata and the deletion prefix of the metadata are stored in the data table to be deleted.

[0014] The metadata deletion method provided in this disclosure can generate a deletion prefix for metadata based on the data identifier of the metadata, and store the deletion prefix and metadata in a data table to be deleted. This allows multiple scanning threads to simultaneously scan the metadata of the target deletion prefix corresponding to each scanning thread, thereby achieving high-concurrency scanning and improving the efficiency of metadata deletion.

[0015] In one embodiment, the method further includes:

[0016] For each metadata in the data table to be deleted, sort them according to the deletion prefix of each metadata;

[0017] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them according to a preset deletion rule to obtain an updated data table to be deleted.

[0018] The metadata deletion method provided in this disclosure can sort metadata according to deletion prefixes and, according to preset deletion rules, prioritize data to be deleted within each deletion prefix. Since metadata with the same deletion prefix is ​​located in the same region, the scanning thread does not need to continue scanning other metadata deletion prefixes after scanning the last metadata with the target deletion prefix. Furthermore, data to be deleted can be scanned by the scanning thread first, thus improving metadata scanning efficiency and meeting user needs in different scenarios.

[0019] In one embodiment, the step of sorting multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix according to a preset deletion rule to obtain an updated data table to be deleted includes:

[0020] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them in ascending order according to the data identifiers corresponding to each metadata entry to obtain the updated data table to be deleted.

[0021] The metadata deletion method provided in this disclosure can sort each metadata in ascending order of data identifier, thus enabling older metadata in each deletion prefix to be deleted first, and quickly clearing metadata that has existed in the system for a long time.

[0022] In one embodiment, the step of sorting multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix according to a preset deletion rule to obtain an updated data table to be deleted includes:

[0023] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them in descending order of the data size corresponding to each metadata entry to obtain a first intermediate sorting result;

[0024] For the first intermediate sorting result, multiple metadata with the same deletion prefix and the same data size are sorted in ascending order according to the storage medium type value corresponding to each metadata to obtain the second intermediate sorting result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium.

[0025] For multiple metadata entries in the second sorting intermediate results that have the same deletion prefix, the same data size, and the same storage medium type value, sort them in ascending order according to the data identifier corresponding to each metadata entry to obtain the updated data table to be deleted.

[0026] The metadata deletion method provided in this disclosure can first sort the metadata according to its data size from largest to smallest, then sort the metadata with the same data size according to its storage medium type value from smallest to largest, and then sort the metadata with the same storage medium type value according to its data identifier from smallest to largest. Therefore, metadata with larger data size, higher access speed of the storage medium, and earlier generation time under each deletion prefix can be deleted first, thereby quickly freeing up system space.

[0027] In one embodiment, the step of sorting multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix according to a preset deletion rule to obtain an updated data table to be deleted includes:

[0028] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, they are sorted in ascending order according to the storage medium type value corresponding to each metadata entry to obtain a third intermediate sorting result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium.

[0029] For the multiple metadata entries in the third intermediate sorting result that have the same deletion prefix and the same storage medium type value, sort them in descending order of the data size corresponding to each metadata entry to obtain the fourth intermediate sorting result;

[0030] For the multiple metadata entries in the fourth sorting intermediate results that have the same deletion prefix, the same storage medium type value, and the same data size, sort them in ascending order according to the data identifier corresponding to each metadata entry to obtain the updated data table to be deleted.

[0031] The metadata deletion method provided in this disclosure can prioritize sorting metadata by storage medium type value from smallest to largest, then sort metadata with the same storage medium type value by data size from largest to smallest, and finally sort metadata with the same data size by data identifier from smallest to largest. Therefore, metadata with higher storage medium access speed, larger data size, and earlier generation time under each deletion prefix can be deleted first, thereby quickly freeing up space on storage media with higher access speeds.

[0032] In one embodiment, the step of caching the data key-value pair of the target metadata corresponding to the target deletion prefix in the data table to be deleted to the storage space of the scanning thread when the target deletion prefix corresponding to the scanning thread is detected includes:

[0033] If a target deletion prefix corresponding to the scanning thread is detected, the starting position of the target deletion prefix in the data table to be deleted is determined. The starting position is the target position of the target metadata that is sorted first in the data table to be deleted among all the target metadata corresponding to the target deletion prefix.

[0034] Starting from the starting position, the deletion prefix of the metadata is detected sequentially, and when the deletion prefix is ​​the target deletion prefix, the data key value of the target metadata is cached in the storage space of the scanning thread.

[0035] The metadata deletion method provided in this disclosure can determine the starting position of the target deletion prefix when it is detected, and sequentially detect the deletion prefixes of each metadata element from the starting position until the detected deletion prefix is ​​not the target deletion prefix. This disclosure can quickly find the target deletion prefix corresponding to the scanning thread. Since sequential detection and scanning only begin from the starting position of the target deletion prefix when it is detected, it can speed up the detection of the target deletion prefix by the scanning thread.

[0036] In one embodiment, detecting the deletion prefix of each metadata element in the data table to be deleted using at least one scanning thread includes:

[0037] Monitor the number of key-value pairs of data stored in the memory space of each scanning thread;

[0038] For any of the scanning threads, when the number of key values ​​corresponding to the scanning thread is less than the scanning threshold, the deletion prefix of each metadata in the data table to be deleted is detected by the scanning thread.

[0039] The metadata deletion method provided in this disclosure can start detecting deletion prefixes in the data table to be deleted by the scanning thread when the number of key-value pairs in the scanning thread is less than the scanning threshold. That is, the scanning thread will perform a scan at intervals instead of continuously, which can reduce the system resources consumed by the scanning thread.

[0040] In one embodiment, caching the data key value of the target metadata corresponding to the target deletion prefix in the data table to be deleted into the storage space of the scanning thread includes:

[0041] The data key value of the target metadata corresponding to the target deletion prefix in the data table to be deleted is cached in the storage space of the scanning thread until the number of data key values ​​stored in the storage space of the scanning thread reaches the storage threshold, or until the target metadata corresponding to the target deletion prefix does not exist in the data table to be deleted.

[0042] The metadata deletion method provided in this disclosure can set a storage threshold for data keys and, during a scan, replenish the number of keys in the scanning thread's storage space to the storage threshold. This means that each scan can scan a maximum of the storage threshold number of metadata values. This disclosure can limit the maximum number of data keys stored in a scanning thread by using a storage threshold, thereby reducing the system resources consumed by the scanning thread.

[0043] In one embodiment, sending data deletion requests to multiple deletion threads based on data key-value pairs cached in the scanning thread includes:

[0044] Get the expected number of concurrent connections;

[0045] Obtain the current concurrency of multiple deletion threads, and determine the target concurrency based on the expected concurrency and the current concurrency, wherein the current concurrency is the number of deletion threads currently performing deletion operations among the deletion threads;

[0046] Based on the data key-value pairs cached in the scanning thread, corresponding data deletion requests are sent concurrently to several deletion threads targeting the target.

[0047] The metadata deletion method provided in this disclosure can determine the target concurrency based on the expected concurrency and the current concurrency, and submit the target number of data deletion requests to the deletion thread. This disclosure can ensure that, when the number of key-value pairs is sufficient, each deletion thread always performs data deletion according to the expected concurrency, thereby accelerating the metadata deletion rate.

[0048] In one embodiment, obtaining the expected concurrency number includes:

[0049] Obtain the number of system business requests and metadata space utilization;

[0050] The expected concurrency is determined based on the number of system service requests and the metadata space utilization rate, wherein the expected concurrency is negatively correlated with the number of system service requests and positively correlated with the metadata space utilization rate.

[0051] The metadata deletion method provided in this disclosure can calculate the expected number of concurrent requests in real time based on the number of system business requests and the metadata space utilization rate. Therefore, it can dynamically adjust the metadata deletion rate according to the current business load of the system and the space occupied by the metadata, balancing the system resources occupied by metadata deletion and the need to delete metadata to release storage space.

[0052] Secondly, this application also provides a metadata deletion device. The device includes:

[0053] The detection module is used to detect the deletion prefix of each metadata in the data table to be deleted through at least one scanning thread;

[0054] The caching module is used to cache the data key value of the target metadata corresponding to the target deletion prefix in the data table to be deleted into the storage space of the scanning thread when the target deletion prefix corresponding to the scanning thread is detected for any of the scanning threads.

[0055] The sending module is used to send data deletion requests to multiple deletion threads based on the data key values ​​cached in the scanning thread, so that the deletion threads delete the target metadata corresponding to the data key value from the data table to be deleted according to the data key value.

[0056] In one embodiment, the device further includes:

[0057] A generation module is configured to, upon receiving a deletion instruction for the metadata, generate the deletion prefix of the metadata based on the data identifier of the metadata;

[0058] A storage module is used to store the metadata and the deletion prefix of the metadata in the data table to be deleted.

[0059] In one embodiment, the device further includes:

[0060] The first sorting module is used to sort the metadata in the data table to be deleted according to the deletion prefix of each metadata.

[0061] The second sorting module is used to sort multiple metadata entries in the data table to be deleted that have the same deletion prefix according to a preset deletion rule, so as to obtain the updated data table to be deleted.

[0062] In one embodiment, the second sorting module is further configured to:

[0063] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them in ascending order according to the data identifiers corresponding to each metadata entry to obtain the updated data table to be deleted.

[0064] In one embodiment, the second sorting module is further configured to:

[0065] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them in descending order of the data size corresponding to each metadata entry to obtain a first intermediate sorting result;

[0066] For the first intermediate sorting result, multiple metadata with the same deletion prefix and the same data size are sorted in ascending order according to the storage medium type value corresponding to each metadata to obtain the second intermediate sorting result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium.

[0067] For multiple metadata entries in the second sorting intermediate results that have the same deletion prefix, the same data size, and the same storage medium type value, sort them in ascending order according to the data identifier corresponding to each metadata entry to obtain the updated data table to be deleted.

[0068] In one embodiment, the second sorting module is further configured to:

[0069] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, they are sorted in ascending order according to the storage medium type value corresponding to each metadata entry to obtain a third intermediate sorting result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium.

[0070] For the multiple metadata entries in the third intermediate sorting result that have the same deletion prefix and the same storage medium type value, sort them in descending order of the data size corresponding to each metadata entry to obtain the fourth intermediate sorting result;

[0071] For the multiple metadata entries in the fourth sorting intermediate results that have the same deletion prefix, the same storage medium type value, and the same data size, sort them in ascending order according to the data identifier corresponding to each metadata entry to obtain the updated data table to be deleted.

[0072] In one embodiment, the caching module is further configured to:

[0073] If a target deletion prefix corresponding to the scanning thread is detected, the starting position of the target deletion prefix in the data table to be deleted is determined. The starting position is the target position of the target metadata that is sorted first in the data table to be deleted among all the target metadata corresponding to the target deletion prefix.

[0074] Starting from the starting position, the deletion prefix of the metadata is detected sequentially, and when the deletion prefix is ​​the target deletion prefix, the data key value of the target metadata is cached in the storage space of the scanning thread.

[0075] In one embodiment, the detection module is further configured to:

[0076] Monitor the number of key-value pairs of data stored in the memory space of each scanning thread;

[0077] For any of the scanning threads, when the number of key values ​​corresponding to the scanning thread is less than the scanning threshold, the deletion prefix of each metadata in the data table to be deleted is detected by the scanning thread.

[0078] In one embodiment, the caching module is further configured to:

[0079] The data key value of the target metadata corresponding to the target deletion prefix in the data table to be deleted is cached in the storage space of the scanning thread until the number of data key values ​​stored in the storage space of the scanning thread reaches the storage threshold, or until the target metadata corresponding to the target deletion prefix does not exist in the data table to be deleted.

[0080] In one embodiment, the sending module is further configured to:

[0081] Get the expected number of concurrent connections;

[0082] Obtain the current concurrency of multiple deletion threads, and determine the target concurrency based on the expected concurrency and the current concurrency, wherein the current concurrency is the number of deletion threads currently performing deletion operations among the deletion threads;

[0083] Based on the data key-value pairs cached in the scanning thread, corresponding data deletion requests are sent concurrently to several deletion threads targeting the target.

[0084] In one embodiment, the sending module is further configured to:

[0085] Obtain the number of system business requests and metadata space utilization;

[0086] The expected concurrency is determined based on the number of system service requests and the metadata space utilization rate, wherein the expected concurrency is negatively correlated with the number of system service requests and positively correlated with the metadata space utilization rate.

[0087] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any of the methods described above.

[0088] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements any of the above methods.

[0089] Fifthly, this application also provides a computer program product. The computer program product includes a computer program, which, when executed by a processor, implements any of the methods described above.

[0090] The aforementioned metadata deletion method, apparatus, computer device, and storage medium can simultaneously detect the deletion prefix of each metadata in the data table to be detected by multiple scanning threads. When the deletion prefix matches the target deletion prefix, the data key value of the target metadata is cached in the storage space of the scanning thread, and data deletion requests are sent to multiple deletion threads based on the deletion key value. This allows the deletion threads to find and delete the target metadata from the data table to be deleted based on the data key value. In this embodiment, the deletion prefix of metadata is set, and each scanning thread only scans target metadata with its corresponding target deletion prefix. Therefore, multiple scanning threads can be used concurrently to scan metadata, improving the scanning speed and, consequently, the efficiency of the scanning threads submitting data deletion requests to the deletion threads. Furthermore, by distinguishing between scanning threads and deletion threads, the scanning and deletion processes are decoupled, avoiding mutual interference between scanning and deletion that could lead to inefficiencies, thus improving the efficiency of metadata deletion. Attached Figure Description

[0091] Figure 1 This is a flowchart illustrating a metadata deletion method in one embodiment;

[0092] Figure 2 This is a flowchart illustrating step 102 in one embodiment;

[0093] Figure 3 This is a flowchart illustrating a metadata deletion method in one embodiment;

[0094] Figure 4 This is a flowchart illustrating step 304 in one embodiment;

[0095] Figure 5 This is a flowchart illustrating step 304 in one embodiment;

[0096] Figure 6 This is a flowchart illustrating step 104 in one embodiment;

[0097] Figure 7 This is a flowchart illustrating step 102 in one embodiment;

[0098] Figure 8 This is a flowchart illustrating step 106 in one embodiment;

[0099] Figure 9 This is a flowchart illustrating step 802 in one embodiment;

[0100] Figure 10 This is a schematic diagram illustrating the transfer of metadata to a data table to be deleted in one embodiment;

[0101] Figure 11 This is a schematic diagram of a deletion rule in one embodiment;

[0102] Figure 12 This is a schematic diagram of the scanning thread and the deletion thread in one embodiment;

[0103] Figure 13 This is a schematic diagram of a scanning thread in one embodiment;

[0104] Figure 14 This is a structural block diagram of a metadata deletion device in one embodiment;

[0105] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0106] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0107] Figure 1 This is a flowchart illustrating a metadata deletion method according to an exemplary embodiment. This embodiment uses the application of this method to a terminal as an example for illustration. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0108] In step 102, at least one scanning thread is used to detect the deletion prefix of each metadata in the data table to be deleted.

[0109] In this embodiment of the disclosure, after a user deletes metadata on the front end, the metadata will be moved from a data table accessible on the front end to a data table to be deleted that is inaccessible on the front end. During the movement of metadata, a corresponding deletion prefix can be generated for the metadata. The deletion prefix indicates which scanning thread should scan the metadata (scanning refers to caching the data key-value pair of the metadata). Each scanning thread only scans metadata with the deletion prefix corresponding to its own. For example, if the deletion prefixes are 1, 2, 3, and 4, and scanning thread A has deletion prefixes 1 and 2, while scanning thread B has deletion prefixes 3 and 4, then scanning thread A will only scan metadata with deletion prefixes 1 and 2, and scanning thread B will only scan metadata with deletion prefixes 3 and 4. The specific number of deletion prefixes can be set by those skilled in the art according to actual needs. For example, when more scanning threads are needed, the number of deletion prefixes can be set to be larger so that each scanning thread corresponds to at least one deletion prefix; when fewer scanning threads are needed, the number of deletion prefixes can be set to be smaller.

[0110] Deletion prefixes can be recorded in the data key-value pairs of metadata. When deleting metadata, a scanning thread can retrieve the deletion prefix from the data key-value pairs of each metadata element in the table to be deleted, and determine whether the deletion prefix corresponds to the deletion prefix of the scanning thread.

[0111] In step 104, for any scanning thread, if the target deletion prefix corresponding to the scanning thread is detected, the data key value of the target metadata corresponding to the target deletion prefix in the data table to be deleted is cached in the storage space of the scanning thread.

[0112] In this embodiment of the disclosure, for any scanning thread, the deletion prefix corresponding to the scanning thread is the target deletion prefix of that scanning thread, and the metadata corresponding to the target deletion prefix is ​​the target metadata that the scanning thread needs to scan. When the scanning thread detects its corresponding target deletion prefix, the metadata corresponding to the target deletion prefix is ​​the target metadata. The scanning thread can cache the data key-value of the target metadata in the storage space so that it can submit the data key-value to the deletion thread, allowing the deletion thread to look up the target metadata from the data table to be deleted based on the data key-value.

[0113] Taking the above example, scanning thread A can sequentially detect the deletion prefix of each metadata in the table to be deleted and compare the deletion prefix with the deletion prefix corresponding to scanning thread A. When scanning thread A detects metadata with a deletion prefix of 3 or 4, scanning thread A can skip that metadata and continue to detect the deletion prefix of the next metadata; when it detects metadata with a deletion prefix of 1 or 2, that is, when it detects the target deletion prefix corresponding to scanning thread A, scanning thread A can cache the data key value of the target metadata and continue to detect the deletion prefix of the next metadata.

[0114] In step 106, based on the data key values ​​cached in the scanning thread, data deletion requests are sent to multiple deletion threads so that the deletion threads can delete the target metadata corresponding to the data key values ​​from the data table to be deleted.

[0115] In this embodiment, the scanning thread can simultaneously submit data deletion requests to multiple deletion threads, and simultaneously send the cached data key-value pairs in the scanning thread's storage space to each deletion thread. This allows each deletion thread to find the target metadata corresponding to the data key-value pair in the data table to be deleted, and then delete the target metadata. Since deleting metadata generally also requires deleting the corresponding data in the system, the deletion thread can also find the data corresponding to the target metadata and delete that data as well.

[0116] The metadata deletion method provided in this disclosure can simultaneously detect the deletion prefix of each metadata in the data table to be detected by multiple scanning threads. When the deletion prefix matches the target deletion prefix, the data key value of the target metadata is cached in the storage space of the scanning thread, and data deletion requests are sent to multiple deletion threads based on the deletion key value. This allows the deletion threads to find and delete the target metadata from the data table to be deleted based on the data key value. This disclosure sets a deletion prefix for metadata and ensures that each scanning thread only scans target metadata with its corresponding target deletion prefix. Therefore, multiple scanning threads can be used concurrently to scan metadata, improving the scanning speed and thus the efficiency of the scanning threads submitting data deletion requests to the deletion threads. Furthermore, by distinguishing between scanning threads and deletion threads, the scanning and deletion processes are decoupled, avoiding mutual interference between scanning and deletion that could lead to inefficiencies, thereby improving the efficiency of metadata deletion.

[0117] In one exemplary embodiment, reference is made to Figure 2 As shown, in step 102, before detecting the deletion prefix of each metadata in the data table to be deleted using at least one scanning thread, the above method further includes:

[0118] In step 202, upon receiving a deletion instruction for metadata, a deletion prefix for the metadata is generated based on the data identifier of the metadata.

[0119] In step 204, the metadata and its deletion prefix are stored in the data table to be deleted.

[0120] In this embodiment of the disclosure, the data identifier is a unique identifier for the metadata. When a user deletes metadata on the front end, a deletion prefix for the metadata can be generated based on the data identifier, and the metadata can be stored in the data table to be deleted. The deletion prefix of the metadata can be stored as part of the data key value of the metadata.

[0121] It should be noted that, to ensure load balancing across scanning threads, the number of generated deletion prefixes should be as similar as possible. This disclosure does not specifically limit the method for generating deletion prefixes. For example, deletion prefixes for metadata can be generated using a hash algorithm.

[0122] The metadata deletion method provided in this disclosure can generate a deletion prefix for metadata based on the data identifier of the metadata, and store the deletion prefix and metadata in a data table to be deleted. This allows multiple scanning threads to simultaneously scan the metadata of the target deletion prefix corresponding to each scanning thread, thereby achieving high-concurrency scanning and improving the efficiency of metadata deletion.

[0123] In one exemplary embodiment, reference is made to Figure 3 As shown, the above method also includes:

[0124] In step 302, the metadata in the data table to be deleted is sorted according to the deletion prefix of each metadata.

[0125] In step 304, multiple metadata entries with the same deletion prefix in the data table to be deleted are sorted according to a preset deletion rule to obtain an updated data table to be deleted.

[0126] In this embodiment of the disclosure, the metadata in the data table to be deleted can be arranged according to the deletion prefix, so that metadata with the same deletion prefix are all located in the same area. This embodiment of the disclosure does not specifically limit the order of the deletion prefixes, as long as the metadata with the same deletion prefix are located in the same area: for example, if the deletion prefixes are 1, 2, 3, and 4, the order of the deletion prefixes can be from smallest to largest (i.e., the order is 1, 2, 3, 4), or from largest to smallest (i.e., the order is 4, 3, 2, 1), or sorted according to a preset fixed position (e.g., the preset order is 2, 3, 1, 4), etc.

[0127] For metadata with the same deletion prefix, since the scanning thread will scan the metadata sequentially from front to back, the data that needs to be deleted first can be arranged at the front so that this part of the data can be scanned by the scanning thread first.

[0128] Deletion rules can be set by the user, or several rules can be pre-defined for the user to choose from. For example, the user can set a rule to prioritize deleting larger metadata, meaning metadata with the same deletion prefix is ​​sorted from largest to smallest. When the scanning thread scans the metadata, the first metadata it scans is the largest metadata with that deletion prefix, and its data key-value pair is stored first in the scanning thread's storage space. When the scanning thread submits a data deletion request, this data key-value pair is also submitted first to the deletion thread, ensuring that this metadata is deleted first.

[0129] It should be noted that the parameters upon which deletion rules depend can also be stored in the data keys of the metadata, allowing the metadata to be sorted based on these data keys. For example, if the deletion rule prioritizes deleting larger metadata entries, the size of the metadata can be stored as part of its data key.

[0130] Users can also change the deletion rules during the data deletion process. After changing the deletion rules, the metadata in the table to be deleted can be reordered according to the new deletion rules.

[0131] After the new metadata is stored in the table to be deleted, all data in the table can be re-sorted to determine the correct position of the new metadata and the correct positions of other metadata after the new metadata is moved to its correct position, resulting in an updated table to be deleted. For example, if the deletion prefixes are 1, 2, 3, and 4, and the metadata in the table to be deleted is arranged in ascending order of the deletion prefixes, and the new metadata has a deletion prefix of 1, then the position of the metadata with the deletion prefix 1 that was originally at the end of the table to be deleted is x. Therefore, the position of the new metadata in the table to be deleted can be determined as x+1. Correspondingly, the positions of all metadata with deletion prefixes 2, 3, and 4 in the table to be deleted will be shifted one position to the right. That is, if the metadata with the deletion prefix 2 was originally at position y in the table to be deleted, then its new position is y+1.

[0132] The metadata deletion method provided in this disclosure can sort metadata according to deletion prefixes and, according to preset deletion rules, prioritize data to be deleted within each deletion prefix. Since metadata with the same deletion prefix is ​​located in the same region, the scanning thread does not need to continue scanning other metadata deletion prefixes after scanning the last metadata with the target deletion prefix. Furthermore, data to be deleted can be scanned by the scanning thread first, thus improving metadata scanning efficiency and meeting user needs in different scenarios.

[0133] In an exemplary embodiment, in step 304, multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix are sorted according to a preset deletion rule to obtain an updated data table to be deleted, including:

[0134] For multiple metadata entries in the data table to be deleted that have the same deletion prefix, sort them in ascending order according to the data identifiers corresponding to each metadata entry to obtain the updated data table to be deleted.

[0135] In this embodiment of the disclosure, the deletion rule can be to prioritize deleting older metadata. Since the data identifier of metadata is a unique identifier for the metadata, and the data identifier of later-generated metadata will be larger than that of earlier-generated metadata, the generation order of the metadata can be determined based on the data identifier. Sort the metadata according to their data identifiers from smallest to largest, so that metadata generated earlier can be deleted first.

[0136] The metadata deletion method provided in this disclosure can sort each metadata in ascending order of data identifier, thus enabling older metadata in each deletion prefix to be deleted first, and quickly clearing metadata that has existed in the system for a long time.

[0137] In one exemplary embodiment, reference is made to Figure 4 As shown, in step 304, multiple metadata entries with the same deletion prefix in the data table to be deleted are sorted according to a preset deletion rule to obtain an updated data table to be deleted, including:

[0138] In step 402, for multiple metadata entries in the data table to be deleted that have the same deletion prefix, they are sorted in descending order of the data size corresponding to each metadata entry to obtain the first intermediate sorting result.

[0139] In step 404, for multiple metadata with the same deletion prefix and the same data size in the first sorting intermediate result, they are sorted in ascending order according to the storage medium type value corresponding to each metadata to obtain the second sorting intermediate result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium.

[0140] In step 406, for multiple metadata entries in the second sorting intermediate results that have the same deletion prefix, the same data size, and the same storage medium type value, they are sorted in ascending order according to the data identifiers corresponding to each metadata entry, to obtain the updated data table to be deleted.

[0141] In this embodiment, the deletion rule can prioritize deleting metadata with larger data sizes. When the metadata sizes are the same, the metadata can be further sorted based on the storage medium type value of the storage medium where the metadata is located. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is inversely proportional to the access speed of the storage medium. Sort the metadata according to the storage medium type value from smallest to largest, so that metadata in storage media with higher access speeds can be deleted first, quickly freeing up storage space in storage media with higher access speeds.

[0142] Because metadata size and storage medium type values ​​are not unique, it's possible that after sorting metadata based on these values, metadata with the same sort order may still exist. To avoid multiple metadata entries corresponding to the same sort order, which could lead to earlier-written metadata being overwritten by later-written metadata, the metadata can be sorted again according to its data identifier from smallest to largest, ensuring that each piece of metadata has its own unique sort order.

[0143] The metadata deletion method provided in this disclosure can first sort the metadata according to its data size from largest to smallest, then sort the metadata with the same data size according to its storage medium type value from smallest to largest, and then sort the metadata with the same storage medium type value according to its data identifier from smallest to largest. Therefore, metadata with larger data size, higher access speed of the storage medium, and earlier generation time under each deletion prefix can be deleted first, thereby quickly freeing up system space.

[0144] In one exemplary embodiment, reference is made to Figure 5 As shown, in step 304, multiple metadata entries with the same deletion prefix in the data table to be deleted are sorted according to a preset deletion rule to obtain an updated data table to be deleted, including:

[0145] In step 502, for multiple metadata in the data table to be deleted that have the same deletion prefix, they are sorted in ascending order of the storage medium type value corresponding to each metadata to obtain a third intermediate sorting result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium.

[0146] In step 504, for multiple metadata with the same deletion prefix and the same storage medium type value in the third sorting intermediate result, they are sorted in descending order of the data size corresponding to each metadata to obtain the fourth sorting intermediate result.

[0147] In step 506, for multiple metadata entries in the fourth sorting intermediate results that have the same deletion prefix, the same storage medium type value, and the same data size, they are sorted in ascending order according to the data identifiers corresponding to each metadata entry, to obtain the updated data table to be deleted.

[0148] In this embodiment, the deletion rule can prioritize deleting metadata with smaller storage medium type values, i.e., metadata stored in storage media with higher access speeds. When the storage medium type values ​​of metadata are the same, the metadata can be further sorted based on its data size, so that metadata with larger data sizes in each storage medium can be deleted first, quickly freeing up storage space.

[0149] Similarly, to avoid multiple metadata entries corresponding to the same sorting order after sorting the metadata according to the above rules, the metadata can be sorted again according to the data identifier from smallest to largest, so that each metadata entry has its unique sorting order.

[0150] The metadata deletion method provided in this disclosure can prioritize sorting metadata by storage medium type value from smallest to largest, then sort metadata with the same storage medium type value by data size from largest to smallest, and finally sort metadata with the same data size by data identifier from smallest to largest. Therefore, metadata with higher storage medium access speed, larger data size, and earlier generation time under each deletion prefix can be deleted first, thereby quickly freeing up space on storage media with higher access speeds.

[0151] In one exemplary embodiment, reference is made to Figure 6 As shown, in step 104, when a target deletion prefix corresponding to the scanning thread is detected, the data key-value pair of the target metadata corresponding to the target deletion prefix in the data table to be deleted is cached in the storage space of the scanning thread, including:

[0152] In step 602, if the target deletion prefix corresponding to the scanning thread is detected, the starting position of the target deletion prefix in the data table to be deleted is determined. The starting position is the target position of the target metadata that is sorted first in the data table to be deleted among all the target metadata corresponding to the target deletion prefix.

[0153] In step 604, the deletion prefix of the metadata is detected sequentially starting from the starting position, and when the deletion prefix is ​​the target deletion prefix, the data key value of the target metadata is cached in the storage space of the scanning thread.

[0154] In this embodiment of the disclosure, since the metadata is arranged in the order of deletion prefix in the data table to be deleted, the scanning thread can quickly detect whether the corresponding target deletion prefix exists in the data table to be deleted using a fast search algorithm such as binary search. If the target deletion prefix is ​​detected in the data table to be deleted, the scanning thread can search backwards to the starting position of the target deletion prefix in the data table to be deleted, that is, the target position of the target metadata that is sorted first among all target metadata with the target deletion prefix, and then sequentially detect the deletion prefix of each metadata starting from the starting position.

[0155] When the scanning thread detects that the deletion prefix of metadata matches the target deletion prefix, it indicates that the metadata is the target metadata. Therefore, the data key-value pairs of the target metadata can be cached in the scanning thread's storage space. When the scanning thread detects that the deletion prefix of metadata is not the target deletion prefix, it means that the scanning thread has already scanned all the target metadata. Therefore, it can stop sequential detection and scanning, and restart fast detection of the deletion prefixes of each metadata in the table to be deleted until the target deletion prefix is ​​detected again.

[0156] The metadata deletion method provided in this disclosure can determine the starting position of the target deletion prefix when it is detected, and sequentially detect the deletion prefixes of each metadata element from the starting position until the detected deletion prefix is ​​not the target deletion prefix. This disclosure can quickly find the target deletion prefix corresponding to the scanning thread. Since sequential detection and scanning only begin from the starting position of the target deletion prefix when it is detected, it can speed up the detection of the target deletion prefix by the scanning thread.

[0157] In an exemplary embodiment, in step 102, referring to Figure 7 As shown, at least one scanning thread is used to detect the deletion prefix of each metadata element in the data table to be deleted, including:

[0158] In step 702, the number of key values ​​of data key values ​​stored in the storage space of each scanning thread is monitored.

[0159] In step 704, for any scanning thread, when the number of key values ​​corresponding to the scanning thread is less than the scanning threshold, the deletion prefix of each metadata in the data table to be deleted is detected by the scanning thread.

[0160] In this embodiment of the disclosure, the storage space of the scanning thread may store multiple data key-value pairs that have not yet been submitted to the deletion thread, and the number of key-value pairs is the quantity of the aforementioned data key-value pairs. A scanning threshold can be preset, and the number of key-value pairs in each scanning thread can be continuously monitored. When the number of key-value pairs is greater than or equal to the scanning threshold, the scanning thread can pause the detection of deletion prefixes for each metadata; when the number of key-value pairs is less than the scanning threshold, the detection of deletion prefixes for each metadata can begin.

[0161] The target of a scan thread can be preset, such as scanning a preset number of metadata items or performing a scan for a preset duration. During a scan, the monitoring of key-value pairs can be paused until the scan thread completes the scan.

[0162] The metadata deletion method provided in this disclosure can start detecting deletion prefixes in the data table to be deleted by the scanning thread when the number of key-value pairs in the scanning thread is less than the scanning threshold. That is, the scanning thread will perform a scan at intervals instead of continuously, which can reduce the system resources consumed by the scanning thread.

[0163] In one exemplary embodiment, in step 104, the data key-value pair of the target metadata corresponding to the target deletion prefix in the data table to be deleted is cached in the storage space of the scanning thread, including:

[0164] The data key-value of the target metadata corresponding to the target deletion prefix in the data table to be deleted is cached in the storage space of the scanning thread until the number of data key-values ​​stored in the storage space of the scanning thread reaches the storage threshold, or until the target metadata corresponding to the target deletion prefix does not exist in the data table to be deleted.

[0165] In this embodiment, a storage threshold for data keys can be preset. When a scanning thread performs a scan, it can scan the difference between the storage threshold and the number of keys in the storage space, that is, fill the storage space with the number of data keys to the storage threshold. Alternatively, if the number of target metadata in the data table to be deleted is less than the difference between the storage threshold and the number of keys, all target metadata can be scanned into the scanning thread's storage space to complete the scan.

[0166] The metadata deletion method provided in this disclosure can set a storage threshold for data keys and, during a scan, replenish the number of keys in the scanning thread's storage space to the storage threshold. This means that each scan can scan a maximum of the storage threshold number of metadata values. This disclosure can limit the maximum number of data keys stored in a scanning thread by using a storage threshold, thereby reducing the system resources consumed by the scanning thread.

[0167] In one exemplary embodiment, reference is made to Figure 8 As shown, in step 106, based on the data key-value pairs cached in the scanning thread, data deletion requests are sent to multiple deletion threads, including:

[0168] In step 802, the expected number of concurrent connections is obtained.

[0169] In step 804, the current concurrency of multiple deletion threads is obtained, and the target concurrency is determined based on the expected concurrency and the current concurrency. The current concurrency is the number of deletion threads that are currently performing deletion operations.

[0170] In step 806, based on the data key-value pairs cached in the scanning thread, corresponding data deletion requests are sent to the target concurrent deletion threads respectively.

[0171] In this embodiment of the disclosure, the expected concurrency can be a preset concurrency or a value calculated based on the resources currently available to the system for metadata deletion. The target concurrency can be determined based on the difference between the expected concurrency and the current concurrency. When the expected concurrency is less than or equal to the current concurrency, the scanning thread does not submit data deletion requests to the deletion thread; when the expected concurrency is greater than the current concurrency, the scanning thread can begin submitting data deletion requests to the deletion thread.

[0172] When the number of key-value pairs in the scanning thread's storage space is greater than or equal to the target concurrency level, a target number of data deletion requests can be submitted concurrently to the target number of deletion threads, meaning a portion of the data key-value pairs in the storage space can be submitted to the deletion threads. When the number of key-value pairs in the scanning thread is less than the target concurrency level, a target number of data deletion requests can be submitted concurrently to each deletion thread, meaning all the data key-value pairs in the storage space can be submitted to the deletion threads.

[0173] The metadata deletion method provided in this disclosure can determine the target concurrency based on the expected concurrency and the current concurrency, and submit the target number of data deletion requests to the deletion thread. This disclosure can ensure that, when the number of key-value pairs is sufficient, each deletion thread always performs data deletion according to the expected concurrency, thereby accelerating the metadata deletion rate.

[0174] In one exemplary embodiment, reference is made to Figure 9 As shown, in step 802, obtaining the expected concurrency includes:

[0175] In step 902, the number of system service requests and metadata space utilization rate are obtained;

[0176] In step 904, the expected concurrency is determined based on the number of system service requests and the metadata space utilization rate. The expected concurrency is negatively correlated with the number of system service requests and positively correlated with the metadata space utilization rate.

[0177] In this embodiment of the disclosure, the number of system service requests refers to the number of service requests currently being made by the system's front-end, and the metadata space utilization rate is the proportion of space occupied by metadata in the system's metadata space. Both of these data can be directly obtained from the system.

[0178] The expected concurrency can be determined based on the number of system business requests and the metadata space utilization rate. The more system business requests there are, the fewer resources available for deletable threads to delete metadata; therefore, the number of system business requests should be negatively correlated with the expected concurrency. Conversely, the higher the metadata space utilization rate, the greater the need for quickly deleting metadata to free up storage space; therefore, the metadata space utilization rate should be positively correlated with the expected concurrency. For example, a maximum concurrency can be preset, and the product of the metadata space utilization rate and the maximum concurrency (a first value) can be determined. The product of 1 and the ratio of the number of system business requests to the maximum number of system business requests, and the maximum concurrency (a second value), can also be determined. The expected concurrency can then be determined based on the larger of the first and second values ​​(see Formula (I)).

[0179]

[0180] Where N_calculate is the expected concurrency, used_ is the metadata space utilization rate, N_max is the maximum concurrency, front__num is the number of system business requests, and req_num_ is the maximum number of system business requests.

[0181] The metadata deletion method provided in this disclosure can calculate the expected number of concurrent requests in real time based on the number of system business requests and the metadata space utilization rate. Therefore, it can dynamically adjust the metadata deletion rate according to the current business load of the system and the space occupied by the metadata, balancing the system resources occupied by metadata deletion and the need to delete metadata to release storage space.

[0182] To enable those skilled in the art to better understand the embodiments of this application, the embodiments of this application are described below through specific examples.

[0183] In this embodiment of the disclosure, metadata is stored in a data table based on an LSM (Log-Structured Merge Tree) storage structure. That is, the storage order of metadata in the data table is related to the data key value of the metadata. Each metadata has its unique data key value, and the same data key value means that the metadata is consistent.

[0184] Reference Figure 10As shown, the system contains two data tables: Table 1 is used for front-end business access, and Table 2, the data table to be deleted, stores the metadata to be deleted for back-end metadata deletion. Here, Key represents the data key-value pair, and Value represents the data value. After a user deletes metadata from Table 1, the user can no longer access that metadata. Simultaneously, a deletion prefix can be generated for the metadata, and partial information from the metadata can be extracted and used together with the deletion prefix as the data key-value pair for the metadata in the data table to be deleted. The metadata is then inserted into the data table to be deleted. The deletion prefix can be generated using a hash algorithm based on the metadata seq (i.e., the metadata's data identifier) ​​in the data key-value pair of the metadata in Table 1. Since the metadata seq is the unique identifier of the metadata, and the number of deletion prefixes obtained from the metadata seq using the hash algorithm is roughly equal, obtaining the deletion prefix based on the metadata seq helps to maintain a balanced load across the scanning threads.

[0185] It should be noted that during metadata deletion, the number of deletion prefixes can be increased, but should not be decreased. This is to avoid a situation where, after reducing the number of deletion prefixes and adjusting the target deletion prefixes for each scanning thread, the scanning threads can no longer scan metadata belonging to the old deletion prefixes that are not part of the new deletion prefixes. For example, if the old deletion prefixes are 1, 2, 3, and 4, scanning thread A corresponds to deletion prefixes 1 and 2, and scanning thread B corresponds to deletion prefixes 3 and 4. After reducing the number of deletion prefixes, the new deletion prefixes are 1 and 2, and scanning thread A corresponds to deletion prefix 1, and scanning thread B corresponds to deletion prefix 2. In this case, if there is still data in the table to be deleted that was not deleted and has scanning prefixes 3 and 4, this data will no longer be scanned by the scanning threads and will remain in Table 2.

[0186] The metadata in Table 2 may include the corresponding layout (file layout). The layout describes the data corresponding to the metadata in the system, storing the location of that data on the disk. When deleting metadata from Table 2, the corresponding data in the system can be deleted along with the metadata based on its layout.

[0187] After deleting metadata and saving it in the table to be deleted, the metadata can be sorted. Since the scanning thread scans metadata sequentially from front to back, data that needs to be deleted first can be prioritized for scanning. (See also...) Figure 11As shown, three deletion rules can be preset: priority deletion of old metadata, priority release of system data space, and priority release of high-speed storage device space. Regardless of the deletion rule selected, the first step of sorting is to sort each metadata in ascending order based on its deletion prefix, so that metadata with the same deletion prefix are arranged together. After this, if priority deletion of old metadata is selected, the second and final step of sorting is to arrange the metadata with smaller sequence numbers, i.e., the metadata uploaded by the user earlier, first. If priority release of system data space is selected, the second step is to arrange the metadata with larger data size first, and the third step is to arrange the metadata with smaller storage medium type values ​​first, so that the system data space can be quickly released and reused. If priority release of high-speed storage device space is selected, the second step is to arrange the metadata with smaller storage medium type values ​​first, and the third step is to arrange the metadata with larger data size first, so as to prioritize the release of storage space of high-speed storage devices (such as NVMe (Non-Volatile Memoryexpress) devices). To avoid different metadata being sorted in the same order, a fourth step can be added when selecting to prioritize deleting older metadata or releasing high-speed storage space. This step sorts the metadata by its sequence number (seq), from smallest to largest. The system's default deletion rule can be set to "prioritize releasing system data space," and the deletion priority can be modified at any time during the metadata deletion process.

[0188] Reference Figure 12As shown, during actual metadata deletion, two thread pools exist simultaneously in the background: a scanning thread pool and a deletion thread pool. The number of scanning threads in the scanning thread pool should not exceed the number of deletion prefixes to avoid multiple scanning threads corresponding to the same deletion prefix, or scanning threads becoming idle due to not corresponding to a deletion prefix. The deletion thread pool can include any number of deletion threads. Each scanning thread concurrently scans metadata from the data table to be deleted, and each scanning thread only scans the metadata of the target deletion prefix corresponding to its own. Since the metadata on the data table to be deleted is arranged in the order of deletion prefixes, the scanning threads can quickly detect whether metadata with the target deletion prefix exists on the data table to be deleted using a fast lookup algorithm. When metadata with the target deletion prefix is ​​detected, the scanning threads can also sequentially detect and scan metadata starting from the beginning of the target deletion prefix. When the deletion prefix of the next piece of metadata is not the target deletion prefix, the sequential detection can be stopped, and the fast detection of metadata with the target deletion prefix can be restarted until metadata with the target deletion prefix is ​​detected again. Grouping metadata with the same deletion prefix together allows the scanning thread to scan metadata under the target deletion prefix in sequence without needing to record breakpoints during the scanning process.

[0189] Reference Figure 13 As shown, the scanning thread will perform scans at regular intervals, rather than continuously. Specifically, a scan threshold can be set for the scanning thread, and the number of key-value pairs stored in the scanning thread can be monitored in real time. Detection and scanning will only begin when the number of key-value pairs is less than the scan threshold, and the difference between the storage threshold and the number of key-value pairs will be scanned into the storage space at once. Simultaneously, the scanning thread can also obtain the number of system business requests, metadata space utilization, and current concurrency. Based on the number of system business requests and metadata space utilization, it determines the expected concurrency. When the expected concurrency is greater than the current concurrency, it submits a data deletion request (the difference between the expected and current concurrency) to the deletion thread. The deletion thread will search for the metadata corresponding to the data key-value pair in the data table to be deleted, and delete the metadata and the data corresponding to that metadata. The current concurrency will increment by 1 for each data deletion request received by the deletion thread; the current concurrency will decrement by 1 after each data deletion request is processed.

[0190] The metadata deletion method provided in this disclosure achieves an efficient, fair, and balanced concurrent scanning and deletion mechanism by generating a deletion prefix based on the metadata sequence number (seq) and adding the deletion prefix to the data key-value pairs of the metadata. This ensures a relatively even distribution of system resources among different scanning threads, resulting in consistent processing efficiency and eliminating uneven workloads. Furthermore, this disclosure allows for the setting of different deletion rules to meet user needs for metadata deletion in various scenarios. Moreover, it can adaptively adjust the background metadata deletion rate in real time based on the number of system business requests and the current metadata space utilization rate. The entire process is fully automated, achieving a dynamic balance between user experience and metadata and data space reclamation efficiency.

[0191] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0192] Based on the same inventive concept, this application also provides a metadata deletion apparatus for implementing the metadata deletion method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more metadata deletion apparatus embodiments provided below can be found in the limitations of the metadata deletion method described above, and will not be repeated here.

[0193] In one embodiment, such as Figure 14 As shown, a metadata deletion device 1400 is provided, including: a detection module 1402, a caching module 1404, and a sending module 1406, wherein:

[0194] The detection module 1402 is used to detect the deletion prefix of each metadata in the data table to be deleted through at least one scanning thread;

[0195] The caching module 1404 is used to cache the data key value of the target metadata corresponding to the target deletion prefix in the data table to be deleted into the storage space of the scanning thread when the target deletion prefix corresponding to the scanning thread is detected for any of the scanning threads.

[0196] The sending module 1406 is used to send data deletion requests to multiple deletion threads based on the data key values ​​cached in the scanning thread, so that the deletion threads delete the target metadata corresponding to the data key value from the data table to be deleted according to the data key value.

[0197] The metadata deletion apparatus provided in this disclosure can simultaneously detect the deletion prefix of each metadata in a data table to be detected by multiple scanning threads. When the deletion prefix matches the target deletion prefix, the data key value of the target metadata is cached in the storage space of the scanning thread. Data deletion requests are then sent to multiple deletion threads based on the deletion key value, allowing the deletion threads to find and delete the target metadata from the data table to be deleted based on the data key value. This disclosure sets a deletion prefix for metadata and ensures that each scanning thread only scans target metadata with its corresponding target deletion prefix. Therefore, multiple scanning threads can be used concurrently to scan metadata, improving the scanning speed and, consequently, the efficiency of the scanning threads submitting data deletion requests to the deletion threads. Furthermore, by distinguishing between scanning and deletion threads, the scanning and deletion processes are decoupled, preventing mutual interference and resulting in low efficiency. This improves the efficiency of metadata deletion.

[0198] In one embodiment, the device further includes:

[0199] A generation module is configured to, upon receiving a deletion instruction for the metadata, generate the deletion prefix of the metadata based on the data identifier of the metadata;

[0200] A storage module is used to store the metadata and the deletion prefix of the metadata in the data table to be deleted.

[0201] In one embodiment, the device further includes:

[0202] The first sorting module is used to sort the metadata in the data table to be deleted according to the deletion prefix of each metadata.

[0203] The second sorting module is used to sort multiple metadata entries in the data table to be deleted that have the same deletion prefix according to a preset deletion rule, so as to obtain the updated data table to be deleted.

[0204] In one embodiment, the second sorting module is further configured to:

[0205] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them in ascending order according to the data identifiers corresponding to each metadata entry to obtain the updated data table to be deleted.

[0206] In one embodiment, the second sorting module is further configured to:

[0207] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them in descending order of the data size corresponding to each metadata entry to obtain a first intermediate sorting result;

[0208] For the first intermediate sorting result, multiple metadata with the same deletion prefix and the same data size are sorted in ascending order according to the storage medium type value corresponding to each metadata to obtain the second intermediate sorting result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium.

[0209] For multiple metadata entries in the second sorting intermediate results that have the same deletion prefix, the same data size, and the same storage medium type value, sort them in ascending order according to the data identifier corresponding to each metadata entry to obtain the updated data table to be deleted.

[0210] In one embodiment, the second sorting module is further configured to:

[0211] For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, they are sorted in ascending order according to the storage medium type value corresponding to each metadata entry to obtain a third intermediate sorting result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium.

[0212] For the multiple metadata entries in the third intermediate sorting result that have the same deletion prefix and the same storage medium type value, sort them in descending order of the data size corresponding to each metadata entry to obtain the fourth intermediate sorting result;

[0213] For the multiple metadata entries in the fourth sorting intermediate results that have the same deletion prefix, the same storage medium type value, and the same data size, sort them in ascending order according to the data identifier corresponding to each metadata entry to obtain the updated data table to be deleted.

[0214] In one embodiment, the cache module 1404 is further configured to:

[0215] If a target deletion prefix corresponding to the scanning thread is detected, the starting position of the target deletion prefix in the data table to be deleted is determined. The starting position is the target position of the target metadata that is sorted first in the data table to be deleted among all the target metadata corresponding to the target deletion prefix.

[0216] Starting from the starting position, the deletion prefix of the metadata is detected sequentially, and when the deletion prefix is ​​the target deletion prefix, the data key value of the target metadata is cached in the storage space of the scanning thread.

[0217] In one embodiment, the detection module 1402 is further configured to:

[0218] Monitor the number of key-value pairs of data stored in the memory space of each scanning thread;

[0219] For any of the scanning threads, when the number of key values ​​corresponding to the scanning thread is less than the scanning threshold, the deletion prefix of each metadata in the data table to be deleted is detected by the scanning thread.

[0220] In one embodiment, the caching module is further configured to:

[0221] The data key value of the target metadata corresponding to the target deletion prefix in the data table to be deleted is cached in the storage space of the scanning thread until the number of data key values ​​stored in the storage space of the scanning thread reaches the storage threshold, or until the target metadata corresponding to the target deletion prefix does not exist in the data table to be deleted.

[0222] In one embodiment, the sending module 1406 is further configured to:

[0223] Get the expected number of concurrent connections;

[0224] Obtain the current concurrency of multiple deletion threads, and determine the target concurrency based on the expected concurrency and the current concurrency, wherein the current concurrency is the number of deletion threads currently performing deletion operations among the deletion threads;

[0225] Based on the data key-value pairs cached in the scanning thread, corresponding data deletion requests are sent concurrently to several deletion threads targeting the target.

[0226] In one embodiment, the sending module 1406 is further configured to:

[0227] Obtain the number of system business requests and metadata space utilization;

[0228] The expected concurrency is determined based on the number of system service requests and the metadata space utilization rate, wherein the expected concurrency is negatively correlated with the number of system service requests and positively correlated with the metadata space utilization rate.

[0229] The modules in the aforementioned metadata deletion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0230] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 15 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a metadata deletion method.

[0231] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0232] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0233] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0234] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0235] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0236] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0237] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0238] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for deleting metadata, characterized in that, The method includes: At least one scanning thread is used to detect the deletion prefix of each metadata in the data table to be deleted, wherein the deletion prefix of the metadata is generated by a hash algorithm based on the data identifier of the metadata; For any of the scanning threads, if a target deletion prefix corresponding to the scanning thread is detected, the starting position of the target deletion prefix in the data table to be deleted is determined. The starting position is the target position of the target metadata that is sorted first in the data table to be deleted among all the target metadata corresponding to the target deletion prefix. Starting from the starting position, the deletion prefix of the metadata is detected sequentially, and when the deletion prefix is ​​the target deletion prefix, the data key value of the target metadata is cached in the storage space of the scanning thread, or the sequential retrieval is stopped when the deletion prefix is ​​not the target deletion prefix; Based on the data key-values ​​cached in the scanning thread, data deletion requests are sent to multiple deletion threads, so that the deletion threads delete the target metadata corresponding to the data key-values ​​from the data table to be deleted.

2. The method according to claim 1, characterized in that, Before detecting the deletion prefix of each metadata in the data table to be deleted using at least one scanning thread, the method further includes: Upon receiving a deletion instruction for the metadata, the deletion prefix of the metadata is generated based on the data identifier of the metadata; The metadata and the deletion prefix of the metadata are stored in the data table to be deleted.

3. The method according to claim 2, characterized in that, The method further includes: For each metadata in the data table to be deleted, sort them according to the deletion prefix of each metadata; For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them according to a preset deletion rule to obtain an updated data table to be deleted.

4. The method according to claim 3, characterized in that, The step of sorting multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix according to a preset deletion rule to obtain an updated data table to be deleted includes: For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them in ascending order according to the data identifiers corresponding to each metadata entry to obtain the updated data table to be deleted.

5. The method according to claim 3, characterized in that, The step of sorting multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix according to a preset deletion rule to obtain an updated data table to be deleted includes: For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, sort them in descending order of the data size corresponding to each metadata entry to obtain a first intermediate sorting result; For the first intermediate sorting result, multiple metadata with the same deletion prefix and the same data size are sorted in ascending order according to the storage medium type value corresponding to each metadata to obtain the second intermediate sorting result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium. For multiple metadata entries in the second sorting intermediate results that have the same deletion prefix, the same data size, and the same storage medium type value, sort them in ascending order according to the data identifier corresponding to each metadata entry to obtain the updated data table to be deleted.

6. The method according to claim 3, characterized in that, The step of sorting multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix according to a preset deletion rule to obtain an updated data table to be deleted includes: For multiple metadata entries in the data table to be deleted that correspond to the same deletion prefix, they are sorted in ascending order according to the storage medium type value corresponding to each metadata entry to obtain a third intermediate sorting result. The storage medium type value is used to characterize the type of storage medium, and the storage medium type value is negatively correlated with the access rate of the storage medium. For the multiple metadata entries in the third intermediate sorting result that have the same deletion prefix and the same storage medium type value, sort them in descending order of the data size corresponding to each metadata entry to obtain the fourth intermediate sorting result; For the multiple metadata entries in the fourth sorting intermediate results that have the same deletion prefix, the same storage medium type value, and the same data size, sort them in ascending order according to the data identifier corresponding to each metadata entry to obtain the updated data table to be deleted.

7. The method according to claim 1, characterized in that, The process of detecting the deletion prefix of each metadata element in the data table to be deleted using at least one scanning thread includes: Monitor the number of key-value pairs of data stored in the memory space of each scanning thread; For any of the scanning threads, when the number of key values ​​corresponding to the scanning thread is less than the scanning threshold, the deletion prefix of each metadata in the data table to be deleted is detected by the scanning thread.

8. The method according to claim 7, characterized in that, The step of caching the data key-value pairs of the target metadata in the storage space of the scanning thread includes: The data key value of the target metadata corresponding to the target deletion prefix in the data table to be deleted is cached in the storage space of the scanning thread until the number of data key values ​​stored in the storage space of the scanning thread reaches the storage threshold, or until the target metadata corresponding to the target deletion prefix does not exist in the data table to be deleted.

9. The method according to claim 1, characterized in that, The step of sending data deletion requests to multiple deletion threads based on the data key-value pairs cached in the scanning thread includes: Get the expected number of concurrent connections; Obtain the current concurrency of multiple deletion threads, and determine the target concurrency based on the expected concurrency and the current concurrency, wherein the current concurrency is the number of deletion threads currently performing deletion operations among the deletion threads; Based on the data key-value pairs cached in the scanning thread, corresponding data deletion requests are sent concurrently to several deletion threads targeting the target.

10. The method according to claim 9, characterized in that, The process of obtaining the expected concurrency includes: Obtain the number of system business requests and metadata space utilization; The expected concurrency is determined based on the number of system service requests and the metadata space utilization rate, wherein the expected concurrency is negatively correlated with the number of system service requests and positively correlated with the metadata space utilization rate.

11. A metadata deletion device, characterized in that, The device includes: The detection module is used to detect the deletion prefix of each metadata in the data table to be deleted through at least one scanning thread, wherein the deletion prefix of the metadata is generated based on the data identifier of the metadata; The caching module is configured to, for any of the scanning threads, upon detecting a target deletion prefix corresponding to the scanning thread, determine the starting position of the target deletion prefix in the data table to be deleted, wherein the starting position is the target position of the target metadata that is sorted first among all the target metadata corresponding to the target deletion prefix in the data table to be deleted; starting from the starting position, sequentially detect the deletion prefix of the metadata, and when the deletion prefix is ​​the target deletion prefix, cache the data key value of the target metadata in the storage space of the scanning thread, or stop the sequential retrieval when the deletion prefix is ​​not the target deletion prefix; The sending module is used to send data deletion requests to multiple deletion threads based on the data key values ​​cached in the scanning thread, so that the deletion threads delete the target metadata corresponding to the data key value from the data table to be deleted according to the data key value.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images