Data processing method and device, equipment and medium

Abstract

The invention discloses a data processing method and device, equipment and a medium, which are applied to the technical field of storage, and the data processing method comprises the following steps: in a metadata deletion interface of a local storage engine, counting the operation times of metadata deletion operation in a key value database by utilizing a first counter to obtain the deletion operation times, the local storage engine is a local storage engine of the distributed storage system; judging whether the deletion operation frequency is greater than a preset deletion operation frequency threshold value or not; and if the deletion operation frequency is greater than the preset deletion operation frequency threshold, triggering a combination compression operation of the key value database to clean tombstones in the key value database. In this way, tombstones generated by metadata deletion in the key value database can be cleaned in time, and the read-write performance of distributed storage is improved.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a data processing method, device, equipment and medium. Background Art

[0002] In the local storage engine of a distributed storage system, deleting a large amount of data corresponds to deleting a large amount of metadata in the local KV (Key-Value) database. The deletion of a large amount of metadata will form a large number of tombstones in the KV database. These tombstones have little impact on database write operations, but have a significant impact on metadata read and list operations. The corresponding metadata needs to be read from the KV database during the data read and write process, which affects the overall IO (Input / Output) performance of the distributed storage.

[0003] Therefore, how to clean up metadata in a timely manner and delete tombstones generated in the key-value database to improve the read and write performance of distributed storage is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention aims to provide a data processing method, apparatus, device, and medium that can promptly clean up metadata and delete tombstones generated in a key-value database, thereby improving the read and write performance of distributed storage. The specific solution is as follows:

[0005] In a first aspect, the present invention provides a data processing method, comprising:

[0006] In a metadata deletion interface of a local storage engine, a first counter is used to count the number of metadata deletion operations in a key-value database to obtain the number of deletion operations, wherein the local storage engine is a local storage engine of a distributed storage system;

[0007] Determine whether the number of deletion operations is greater than a preset deletion operation number threshold;

[0008] If the number of deletion operations is greater than the preset deletion operation number threshold, a merge and compression operation of the key-value database is triggered to clean up tombstones in the key-value database.

[0009] Optionally, in the metadata deletion interface of the local storage engine, before using the first counter to count the number of metadata deletion operations in the key-value database, the method further includes:

[0010] An atomic counter is added to the data structure of the local storage engine as the first counter.

[0011] Optionally, after triggering the merge and compaction operation of the key-value database to clean up tombstones in the key-value database, the method further includes:

[0012] The first counter is cleared so that the first counter re-counts the number of metadata deletion operations in the key-value database.

[0013] Optionally, also include:

[0014] During the placement group deletion process, the second counter is used to count the number of deleted objects to obtain the number of deleted objects;

[0015] Determine whether the number of deleted objects is greater than a preset deletion object number threshold;

[0016] If the number of deleted objects is greater than the preset threshold number of deleted objects, a merge and compression operation of the key-value database is triggered to clean up tombstones in the key-value database.

[0017] Optionally, during the placement group deletion process, before using the second counter to count the number of deleted objects and obtaining the number of deleted objects, the process further includes:

[0018] An atomic counter is added to the data structure of the placement group to obtain the second counter.

[0019] Optionally, after triggering the merge and compression operation of the key-value database to clear tombstones in the key-value database, the following steps may also be included:

[0020] The second counter is cleared so that the second counter can re-count the number of objects deleted during the placement group deletion process.

[0021] Optionally, during the placement group deletion process, the second counter is used to count the number of deleted objects to obtain the number of deleted objects, including:

[0022] During the placement group deletion process, enumerate the objects in the placement group, and add the number of enumerated objects to the second counter to obtain the number of deleted objects;

[0023] Accordingly, the method further includes:

[0024] Submit a delete transaction to the local storage engine so that the local storage engine deletes the listed objects in the key-value store.

[0025] In a second aspect, the present invention provides a data processing device, comprising:

[0026] a deletion operation count statistics module, configured to use a first counter to count the number of metadata deletion operations in the key-value database in a metadata deletion interface of a local storage engine, to obtain the number of deletion operations, wherein the local storage engine is a local storage engine of a distributed storage system;

[0027] An operation number threshold judgment module is used to judge whether the number of deletion operations is greater than a preset deletion operation number threshold;

[0028] The merging and compression operation triggering module is configured to trigger a merging and compression operation of the key-value database if the number of deletion operations is greater than the preset deletion operation number threshold, so as to clean up tombstones in the key-value database.

[0029] In a third aspect, the present invention provides an electronic device, comprising:

[0030] memory for storing computer programs;

[0031] A processor is used to execute the computer program to implement the steps of the aforementioned data processing method.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the aforementioned data processing method when executed by a processor.

[0033] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of the aforementioned disclosed data processing method when executed by a processor.

[0034] From the above scheme, it can be seen that the present invention provides a data processing method, including: in the metadata deletion interface of the local storage engine, using a first counter to count the number of metadata deletion operations in the key-value database to obtain the number of deletion operations, wherein the local storage engine is the local storage engine of the distributed storage system; judging whether the number of deletion operations is greater than a preset deletion operation number threshold; if the number of deletion operations is greater than the preset deletion operation number threshold, triggering a merge and compression operation of the key-value database to clean up the tombstones in the key-value database.

[0035] As can be seen, the beneficial effect of the present invention lies in: in the metadata deletion interface of the local storage engine, a counter is used to count the number of metadata deletion operations in the key-value database. When the number of deletion operations exceeds a preset deletion operation threshold, a merge and compression operation of the key-value database is triggered to clear the tombstones in the key-value database. In this way, by adding a counter to count the number of metadata deletion operations, and triggering a merge and compression operation to clear the tombstones when the number of deletion operations accumulates to a certain number, the tombstones generated by metadata deletions in the key-value database can be cleared in a timely manner, thereby improving the read and write performance of distributed storage.

[0036] Correspondingly, a data processing device, equipment and medium provided by the present invention also have the above technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 A flow chart of a data processing method provided by an embodiment of the present invention;

[0039] Figure 2 A sub-flowchart of a data processing method provided by an embodiment of the present invention;

[0040] Figure 3 A sub-flowchart of another data processing method provided by an embodiment of the present invention;

[0041] Figure 4 A schematic structural diagram of a data processing device provided in an embodiment of the present invention;

[0042] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] The terms "including" and "having," as used in the present description and accompanying drawings, and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements and may include steps or elements that are not listed.

[0045] The present invention relates to a design method for solving the problem of slow enumeration after a large number of deletions based on distributed storage in the field of data storage.

[0046] With the continuous development of information technology, data has become increasingly valued as a valuable resource. How to quickly process data resources and achieve desired results has become a key issue in transforming them from resources to assets. People generate data in various activities in their work and daily lives. Collecting and analyzing this data can yield valuable insights, transforming resources into assets, thereby driving the rapid development of big data and high-performance computing. Business operations, traffic management, and public security management also generate vast amounts of data. This data often has specific storage cycles, requiring ready retrieval and accessibility within those cycles. Driven by the generation of massive amounts of data, data storage, as a core element of data resources, has also experienced rapid growth. Traditional network storage systems utilize centralized storage servers to store all data. These servers become a performance bottleneck and a key concern for reliability and security, failing to meet the needs of large-scale storage applications. Distributed network storage systems, with their scalable architecture, not only improve reliability, availability, and access efficiency, but also facilitate scalability, gaining increasing acceptance within enterprises. Distributed storage systems typically exist as storage server clusters. A typical storage server cluster consists of 10 storage server nodes, and the largest storage server cluster currently consists of 1024 storage server nodes, providing high-performance, massive data storage. Distributed storage servers can be categorized by storage media, including mechanical disk storage, hybrid flash storage, and all-flash storage. The choice is generally determined based on the customer's business model and storage purchase cost.

[0047] Over time, massive distributed storage inevitably experiences large amounts of data deletion within a short period of time. This can occur, for example, when deleting directories containing numerous files in file scenarios, deleting volumes filled with data in block scenarios, or rebuilding after a cluster failure in general scenarios. In local storage engines based on Bluestore (a transactional local log file system), deleting large amounts of data corresponds to deleting a large amount of metadata in the local key-value database (KV) database. This large amount of metadata deletion creates a large number of tombstones in the KV database. These tombstones have little impact on database write operations, but significantly affect metadata read and enumeration operations. Both data read and write operations require reading the corresponding metadata from the KV database, impacting the overall I / O performance of distributed storage.

[0048] In distributed storage, the general logic is to slowly merge and process tombstones using the KV database's own merge and compression mechanism. However, this mechanism relies on the injection of front-end write traffic, but this coupling significantly impacts performance. If distributed storage experiences a period of concentrated read traffic and minimal write traffic, the KV database will not trigger the merge and compression mechanism, thus prolonging the impact of tombstones on service performance. Modifying the parameters of the KV database's merge and compression mechanism to increase the frequency of triggering the merge and compression mechanism and the amount of data compressed in a single merge and compression is one solution. By performing more frequent and slightly larger merge and compression operations, the impact of tombstones on I / O performance can be shortened. However, this approach affects all database write operations. Deletions are also considered database write operations and are implemented by consuming the overall I / O performance of the distributed storage. While this approach does not significantly impact overall distributed storage I / O performance, the frequent merge and compression consumes more computing and I / O resources on storage nodes, which in turn reduces the I / O performance of the distributed storage cluster.

[0049] To this end, the present invention provides a data processing solution that can clean up metadata in a timely manner and delete tombstones generated in a key-value database, thereby improving the read and write performance of distributed storage.

[0050] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0051] Next, a data processing method provided by an embodiment of the present invention is described in detail. Figure 1 A flow chart of a data processing method provided in an embodiment of the present invention, the data processing method comprising:

[0052] Step S11: In the metadata deletion interface of the local storage engine, a first counter is used to count the number of metadata deletion operations in the key-value database to obtain the number of deletion operations, wherein the local storage engine is a local storage engine of the distributed storage system.

[0053] In an embodiment of the present invention, in the metadata deletion interface of the local storage engine, before using the first counter to count the number of metadata deletion operations in the key-value database, it also includes: adding an atomic counter to the data structure of the local storage engine as the first counter. That is, in an embodiment of the present invention, the first counter is an atomic counter, and adding an atomic counter to the data structure of the local storage engine that records the number of metadata deletion operations during customer business processing can avoid the problem of unexpected kv database merging and compression caused by counting confusion caused by multi-threaded processing. In a distributed storage system, the local storage engine is one of the core components and is responsible for actual data storage and management.

[0054] Step S12: Determine whether the number of deletion operations is greater than a preset deletion operation number threshold.

[0055] The preset deletion operation number threshold may be an empirical value, such as 100,000.

[0056] Step S13: If the number of deletion operations is greater than the preset deletion operation number threshold, a merge and compression operation of the key-value database is triggered to clean up tombstones in the key-value database.

[0057] That is, in the embodiment of the present invention, the asynchronous merging and compression mechanism of the key-value database is actively triggered to clean up the tombstones in the key-value database.

[0058] If the number of deletion operations is less than or equal to the preset deletion operation threshold, the merging and compression operation of the key-value database is not triggered, and the first counter is used to continue counting.

[0059] Furthermore, after triggering the merge and compaction operation of the key-value database to clear tombstones in the key-value database, the embodiment of the present invention further includes: clearing the first counter so that the first counter re-counts the number of metadata deletion operations in the key-value database. In this way, clearing the first counter allows the first counter to restart counting.

[0060] Furthermore, in an optional embodiment, the present invention may also be implemented by using a second counter to count the number of deleted objects during the placement group deletion process to obtain the number of deleted objects; judging whether the number of deleted objects is greater than a preset deletion object number threshold; if the number of deleted objects is greater than the preset deletion object number threshold, triggering a merge and compression operation of the key-value database to clean up the tombstones in the key-value database.

[0061] The preset threshold for the number of deleted objects is smaller than the preset threshold for the number of deletion operations. For example, if the preset threshold for the number of deletion operations is 100,000, the preset threshold for the number of deleted objects is 30,000. Placement group deletion occurs in two situations: when the entire storage pool is deleted, or when a storage cluster fails and recovers. In both cases, the KV database must be restored to a healthy and usable state as quickly as possible. Therefore, the preset threshold for the number of deleted objects is smaller than the preset threshold for the number of deletion operations.

[0062] Furthermore, during the placement group deletion process, before obtaining the number of deleted objects by using the second counter to count the number of deleted objects, the method further includes: adding an atomic counter to the data structure of the placement group to obtain the second counter.

[0063] In addition, after triggering the merge and compression operation of the key-value database to clear tombstones in the key-value database, the method may further include: clearing the second counter so that the second counter re-counts the number of objects deleted during the placement group deletion process.

[0064] In an optional embodiment, during the placement group deletion process, a second counter is used to count the number of deleted objects to obtain the number of deleted objects, including: during the placement group deletion process, listing the objects in the placement group, adding the number of listed objects to the second counter, and obtaining the number of deleted objects; accordingly, the method also includes: submitting a deletion transaction to the local storage engine so that the local storage engine deletes the listed objects in the key-value database.

[0065] In an embodiment of the present invention, a storage pool includes multiple OSDs (Object Storage Devices), each of which includes multiple placement groups. In this embodiment of the present invention, during the process of deleting a placement group, objects in the placement group can be enumerated based on a key-value database. A loop enumeration process is used, with a preset number of objects enumerated each time, placed in a preset table, and the starting position of the next enumeration recorded in the preset table. The number of enumerated objects is incremented in a second counter, and a deletion transaction is submitted to a local storage engine, causing the local storage engine to delete the enumerated objects from the key-value database. A determination is then made as to whether the number of deleted objects exceeds a preset threshold for the number of deleted objects. If the number of deleted objects exceeds the threshold, a merge-compact operation is triggered in the key-value database to clear tombstones in the key-value database and reset the second counter. If the number of deleted objects does not exceed the threshold for the number of deleted objects, a merge-compact operation is not triggered in the key-value database. The loop enumeration process continues, enumerating objects based on the starting position of the next enumeration recorded in the preset table, until all objects in each placement group are enumerated. Metadata in the key-value database is object metadata. In this way, tombstones can be cleaned up in time to ensure the rapid reconstruction of the storage pool.

[0066] As can be seen, in the metadata deletion interface of the local storage engine, an embodiment of the present invention utilizes a counter to count the number of metadata deletion operations in the key-value database. When the number of deletion operations exceeds a preset deletion operation threshold, a merge and compression operation of the key-value database is triggered to clean up the tombstones in the key-value database. Thus, by adding a counter to count the number of metadata deletion operations and triggering a merge and compression operation to clean up the tombstones when a certain number of deletion operations accumulates, tombstones generated by metadata deletions in the key-value database can be cleaned up in a timely manner, thereby improving the read and write performance of distributed storage.

[0067] Furthermore, the present invention proposes a data processing solution that can solve the problem of slow enumeration after a large number of deletions based on distributed storage. According to the characteristics of distributed storage, the root cause of the generation of tombstones in kv data is that a large number of deletion operations are generated within a complete merge and compression cycle, resulting in a large amount of invalid data in the kv database, and the reading and enumeration operations of the kv database cannot directly distinguish which data are invalid, resulting in a large amount of additional overhead, thereby affecting the read and write performance. The embodiment of the present invention solves the problem of slow enumeration after a large number of deletions based on distributed storage, and records the number of metadata deletion operations in the metadata deletion interface of the local storage engine as atomic operations to avoid the problem of confusion in counting due to multi-threaded processing. When the metadata deletion operation count reaches the set threshold 1 (that is, the preset deletion operation number threshold), the purpose of clearing tombstones is achieved by triggering the merge and compression mechanism of the asynchronous kv database. After the asynchronous merge and compression mechanism of the kv database is triggered, the atomic count of the metadata deletion operation is cleared, and the next round of metadata deletion operation count is started. Similarly, an atomic counter is added to the PG (Placement Group) data structure to record the number of objects deleted during PG deletion. When the counter reaches a threshold of 2 (the preset threshold for object deletion), the asynchronous merge and compaction mechanism of the KV database is triggered. After the asynchronous merge and compaction mechanism is triggered, the atomic counter is cleared. Considering real-world scenarios, PG deletion occurs in two situations: the entire storage pool is deleted, and the storage cluster fails and recovers. In both cases, the KV database must be restored to a healthy and usable state as quickly as possible. Therefore, threshold 2 must be smaller than threshold 1. Therefore, this embodiment of the present invention adds an atomic counter to the local storage engine and PG data structure to count deletions. When metadata is deleted, the atomic counter is incremented by 1, and the number of objects deleted during PG deletion is added to the atomic counter. If the atomic counter reaches the threshold, the asynchronous merge and compaction mechanism of the KV database is triggered. The atomic counter is then cleared, and the next counting cycle begins.

[0068] For further information, see Figure 2 As shown, Figure 2A sub-flowchart of a data processing method provided by an embodiment of the present invention. An atomic counter that records the number of metadata deletion operations during customer business processing is added to the data structure of the local storage engine to avoid counting confusion caused by multi-threaded processing, which may lead to unexpected KV database merging and compression problems. In the metadata deletion interface, the original counter is used to count the number of deletion operations. When it is determined that the atomic counter reaches the threshold of 1, the asynchronous merging and compression mechanism of the KV database is actively triggered. The atomic counter is cleared to enter the next statistical cycle.

[0069] For further information, see Figure 3 As shown, Figure 3 This is a sub-flowchart of another data processing method provided by an embodiment of the present invention. An atomic counter is added to the PG data structure to record the number of objects deleted during the PG deletion process. This atomic counter is used to record the number of deleted objects during the PG deletion process. When it is determined that the atomic counter reaches a threshold of 2, the asynchronous merge and compression mechanism of the KV database is actively triggered. The atomic counter is cleared to enter the next statistical cycle.

[0070] That is, after a large number of deletions are made to the distributed storage, the KV database merge and compression mechanism is actively triggered. The embodiment of the present invention adds an atomic counter to count the number of metadata deleted by customer services and the number of objects deleted during the PG deletion process. After determining that the atomic counter reaches the threshold, the asynchronous merge and compression mechanism of the KV database is actively triggered, and then the atomic counter is cleared to enter the next statistical cycle.

[0071] This approach counts the number of customer business metadata deletions and the number of objects deleted during the PG deletion process. Once the count reaches a threshold, the asynchronous merge and compression mechanism in the KV database is triggered, and the atomic counter is then cleared. This avoids impacting normal customer business I / O performance, accelerates the cleanup of tombstones generated in the KV database by large amounts of deleted metadata, and resolves the issue of degraded read and write performance in the distributed storage cluster after large-scale deletions. This improves the read and write performance of distributed storage, enhances product competitiveness, and enhances user experience.

[0072] See also Figure 4 As shown, Figure 4 A schematic structural diagram of a data processing device provided in an embodiment of the present invention, the data processing device comprising:

[0073] a deletion operation count counting module 41 for counting the number of metadata deletion operations in the key-value database using a first counter in a metadata deletion interface of a local storage engine, to obtain the number of deletion operations, wherein the local storage engine is a local storage engine of a distributed storage system;

[0074] An operation times threshold determination module 42 is configured to determine whether the number of deletion operations is greater than a preset deletion operation times threshold;

[0075] The merge and compress operation triggering module 43 is configured to trigger a merge and compress operation of the key-value database if the number of deletion operations is greater than the preset deletion operation number threshold, so as to clean up tombstones in the key-value database.

[0076] In an optional embodiment, an embodiment of the present invention also includes a first counter adding module, which is used to add an atomic counter in the data structure of the local storage engine as the first counter before using the first counter to count the number of metadata deletion operations in the key-value database in the metadata deletion interface of the local storage engine.

[0077] In an optional embodiment, the embodiment of the present invention further includes a first counter clearing module, which is used to clear the first counter after triggering a merge and compression operation of the key-value database to clean up the tombstones in the key-value database, so that the first counter re-counts the number of metadata deletion operations in the key-value database.

[0078] In an optional implementation manner, the embodiment of the present invention further includes:

[0079] a deleted object number counting module, configured to count the number of deleted objects using the second counter during the placement group deletion process to obtain the number of deleted objects;

[0080] An object number threshold determination module, configured to determine whether the number of deleted objects is greater than a preset deletion object number threshold;

[0081] Correspondingly, the merge and compression operation triggering module 43 is further configured to trigger a merge and compression operation of the key-value database if the number of deleted objects is greater than the preset threshold number of deleted objects, so as to clean up tombstones in the key-value database.

[0082] In an optional implementation, the embodiment of the present invention further includes a second counter adding module, which is used to use the second counter to count the number of deleted objects during the placement group deletion process, and before obtaining the number of deleted objects, add an atomic counter to the data structure of the placement group to obtain the second counter.

[0083] In an optional embodiment, the embodiment of the present invention further includes a second counter clearing module, which is used to clear the second counter after triggering a merge and compression operation of the key-value database to clear the tombstones in the key-value database, so that the second counter can re-count the number of objects deleted during the placement group deletion process.

[0084] In an optional embodiment, the deleted object count module may be configured to: during a placement group deletion process, list objects in the placement group, add the number of listed objects to the second counter, and obtain the number of deleted objects;

[0085] Accordingly, the device is also used for:

[0086] Submit a delete transaction to the local storage engine so that the local storage engine deletes the listed objects in the key-value store.

[0087] As can be seen, in the metadata deletion interface of the local storage engine, an embodiment of the present invention utilizes a counter to count the number of metadata deletion operations in the key-value database. When the number of deletion operations exceeds a preset deletion operation threshold, a merge and compression operation of the key-value database is triggered to clean up the tombstones in the key-value database. Thus, by adding a counter to count the number of metadata deletion operations and triggering a merge and compression operation to clean up the tombstones when a certain number of deletion operations accumulates, tombstones generated by metadata deletions in the key-value database can be cleaned up in a timely manner, thereby improving the read and write performance of distributed storage.

[0088] Figure 4 The description of the features in the corresponding embodiment can be found in Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0089] Figure 5 A structural diagram of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device includes: a memory 50 for storing computer programs;

[0090] The processor 51 is configured to implement the steps of the data processing method in the above embodiment when executing a computer program.

[0091] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: in the metadata deletion interface of the local storage engine, use a first counter to count the number of metadata deletion operations in the key-value database to obtain the number of deletion operations, wherein the local storage engine is a local storage engine of a distributed storage system; determine whether the number of deletion operations is greater than a preset deletion operation number threshold; if the number of deletion operations is greater than the preset deletion operation number threshold, trigger a merge and compression operation of the key-value database to clean up the tombstones in the key-value database.

[0092] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: in the metadata deletion interface of the local storage engine, before using the first counter to count the number of metadata deletion operations in the key-value database, an atomic counter is added to the data structure of the local storage engine as the first counter.

[0093] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: after triggering a merge and compression operation of the key-value database to clean up tombstones in the key-value database, clearing the first counter so that the first counter re-counts the number of metadata deletion operations in the key-value database.

[0094] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: during the placement group deletion process, using a second counter to count the number of deleted objects to obtain the number of deleted objects; determining whether the number of deleted objects is greater than a preset deletion object number threshold; if the number of deleted objects is greater than the preset deletion object number threshold, triggering a merge and compression operation of the key-value database to clean up tombstones in the key-value database.

[0095] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: during the placement group deletion process, use the second counter to count the number of deleted objects, and before obtaining the number of deleted objects, add an atomic counter to the data structure of the placement group to obtain the second counter.

[0096] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: after triggering the merge and compression operation of the key-value database to clear the tombstones in the key-value database, the second counter is cleared so that the second counter re-counts the number of objects deleted during the placement group deletion process.

[0097] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: during the placement group deletion process, list the objects in the placement group, add the number of listed objects to the second counter, and obtain the number of deleted objects; submit the deletion transaction to the local storage engine so that the local storage engine deletes the listed objects in the key-value database.

[0098] The electronic device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0099] The processor 51 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 51 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 51 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 51 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing content required to be displayed on the display screen. In some embodiments, the processor 51 may also include an artificial intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0100] The memory 50 may include one or more computer-readable storage media, which may be non-transitory. The memory 50 may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 50 is at least used to store the following computer program 501, wherein, after the computer program is loaded and executed by the processor 51, it can implement the relevant steps of the data processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 502 may include Windows, Unix, Linux, etc. Data 503 may include but is not limited to metadata, etc.

[0101] In some embodiments, the electronic device may further include a display screen 52 , an input / output interface 53 , a communication interface 54 , a power supply 55 , and a communication bus 56 .

[0102] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure.

[0103] It is understood that if the data processing method in the above-mentioned embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, a magnetic disk, or an optical disk, and other media that can store program code.

[0104] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0105] In this embodiment, the computer program executed by the processor can specifically implement the following steps: in the metadata deletion interface of the local storage engine, using a first counter to count the number of metadata deletion operations in the key-value database to obtain the number of deletion operations, wherein the local storage engine is a local storage engine of a distributed storage system; determining whether the number of deletion operations is greater than a preset deletion operation number threshold; if the number of deletion operations is greater than the preset deletion operation number threshold, triggering a merge and compression operation of the key-value database to clean up the tombstones in the key-value database.

[0106] In this embodiment, the computer program executed by the processor can specifically implement the following steps: in the metadata deletion interface of the local storage engine, before using the first counter to count the number of metadata deletion operations in the key-value database, an atomic counter is added to the data structure of the local storage engine as the first counter.

[0107] In this embodiment, the computer program executed by the processor can specifically implement the following steps: after triggering a merge and compression operation of the key-value database to clean up tombstones in the key-value database, clearing the first counter so that the first counter re-counts the number of metadata deletion operations in the key-value database.

[0108] In this embodiment, the computer program executed by the processor can specifically implement the following steps: during the placement group deletion process, using a second counter to count the number of deleted objects to obtain the number of deleted objects; determining whether the number of deleted objects is greater than a preset deletion object number threshold; if the number of deleted objects is greater than the preset deletion object number threshold, triggering a merge and compression operation of the key-value database to clean up tombstones in the key-value database.

[0109] In this embodiment, the computer program executed by the processor can specifically implement the following steps: during the placement group deletion process, using the second counter to count the number of deleted objects, before obtaining the number of deleted objects, adding an atomic counter to the data structure of the placement group to obtain the second counter.

[0110] In this embodiment, the computer program executed by the processor can specifically implement the following steps: after triggering the merge and compression operation of the key-value database to clear the tombstones in the key-value database, the second counter is cleared so that the second counter re-counts the number of objects deleted during the placement group deletion process.

[0111] In this embodiment, the computer program executed by the processor can specifically implement the following steps: during the placement group deletion process, list the objects in the placement group, add the number of listed objects to the second counter, and obtain the number of deleted objects; submit the deletion transaction to the local storage engine so that the local storage engine deletes the listed objects in the key-value database.

[0112] A computer program product provided by an embodiment of the present invention is introduced below. The computer program product described below can be referenced with other embodiments described herein.

[0113] A computer program product comprises a computer program / instruction, which implements the steps of the aforementioned data processing method when executed by a processor.

[0114] The above describes in detail a data processing method, apparatus, device, and medium provided by embodiments of the present invention. The various embodiments are described in a progressive manner throughout this specification, with each embodiment focusing on its differences from other embodiments. Reference can be made to the descriptions of the embodiments for similar or identical parts. The apparatus disclosed in the embodiments corresponds to the method disclosed in the embodiments, so the description is relatively brief. For relevant details, refer to the description of the method.

[0115] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0116] The above is a detailed introduction to a data processing method, device, equipment and medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: In a metadata deletion interface of a local storage engine, a first counter is used to count the number of metadata deletion operations in a key-value database to obtain the number of deletion operations, wherein the local storage engine is a local storage engine of a distributed storage system; Determine whether the number of deletion operations is greater than a preset deletion operation number threshold; If the number of deletion operations is greater than the preset deletion operation number threshold, a merge and compression operation of the key-value database is triggered to clean up tombstones in the key-value database.

2. The data processing method according to claim 1, wherein: In the metadata deletion interface of the local storage engine, before using the first counter to count the number of metadata deletion operations in the key-value database, the method further includes: An atomic counter is added to the data structure of the local storage engine as the first counter.

3. The data processing method according to claim 1, wherein: After triggering the merge and compaction operation of the key-value database to clean up tombstones in the key-value database, the method further includes: The first counter is cleared so that the first counter re-counts the number of metadata deletion operations in the key-value database.

4. The data processing method according to any one of claims 1 to 3, characterized in that: Also includes: During the placement group deletion process, the second counter is used to count the number of deleted objects to obtain the number of deleted objects; Determine whether the number of deleted objects is greater than a preset deletion object number threshold; If the number of deleted objects is greater than the preset threshold number of deleted objects, a merge and compression operation of the key-value database is triggered to clean up tombstones in the key-value database.

5. The data processing method according to claim 4, characterized in that: During the placement group deletion process, the second counter is used to count the number of deleted objects. Before obtaining the number of deleted objects, the following steps are also included: An atomic counter is added to the data structure of the placement group to obtain the second counter.

6. The data processing method according to claim 4, characterized in that: After triggering the merge and compression operation of the key-value database to clear tombstones in the key-value database, the method may further include: The second counter is cleared so that the second counter can re-count the number of objects deleted during the placement group deletion process.

7. The data processing method according to claim 4, characterized in that: During the placement group deletion process, the second counter is used to count the number of deleted objects to obtain the number of deleted objects, including: During the placement group deletion process, enumerate the objects in the placement group, and add the number of enumerated objects to the second counter to obtain the number of deleted objects; Accordingly, the method further includes: Submit a delete transaction to the local storage engine so that the local storage engine deletes the listed objects in the key-value store.

8. A data processing device, characterized in that: include: a deletion operation count statistics module, configured to use a first counter to count the number of metadata deletion operations in the key-value database in a metadata deletion interface of a local storage engine, to obtain the number of deletion operations, wherein the local storage engine is a local storage engine of a distributed storage system; An operation number threshold judgment module is used to judge whether the number of deletion operations is greater than a preset deletion operation number threshold; The merging and compression operation triggering module is configured to trigger a merging and compression operation of the key-value database if the number of deletion operations is greater than the preset deletion operation number threshold, so as to clean up tombstones in the key-value database.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 7.

Images