Data processing

Abstract

One or more embodiments in the present description relate to the technical field of databases. Provided are a data processing method and apparatus. The method is applied to a database based on a combination of a log structure and a tree storage structure. In the solution provided in the present description, the method comprises: acquiring the volume of data to be deleted recorded in an incremental sorted string table (SSTable) of a target partition; and if the volume of said data recorded in the incremental SSTable reaches a first volume threshold value, combining the incremental SSTable into a baseline SSTable of the target partition, so as to delete said data. By means of the solution provided in the present description, data to be deleted can be deleted from a target partition in a timely manner, thereby optimizing the query performance of a database, and reducing unnecessary occupancy of a disk space by said data.

Description

Data processing Technical Field

[0001] One or more embodiments of this specification relate to the field of database technology, and more particularly, to a data processing method and apparatus. Background Art

[0002] In a database based on a log-structured merge tree storage structure, in order to improve the write performance of the database, when a data deletion instruction is executed, the corresponding data is not immediately deleted from the disk. Instead, a tombstone is written to mark the deleted data. However, this will have a certain impact on the query performance of the database, and invalid data with tombstones will take up additional disk space. Summary of the Invention

[0003] In view of this, one or more embodiments of this specification provide a data processing method and apparatus.

[0004] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions.

[0005] According to the first aspect of one or more embodiments of the present specification, a data processing method is proposed, which is applied to a database based on a log-structured merge tree storage structure, the method comprising: obtaining the number of data to be deleted recorded in the incremental ordered string table SSTable of the target partition; if the number of data to be deleted recorded in the incremental SSTable reaches a first quantity threshold, merging the incremental SSTable into the baseline SSTable of the target partition to delete the data to be deleted.

[0006] According to the second aspect of one or more embodiments of the present specification, a data processing device is proposed, which is applied to a database based on a log-structured merge tree storage structure, the device comprising: an acquisition module for acquiring the number of data to be deleted recorded in the incremental ordered string table SSTable of the target partition; a merging module for merging the incremental SSTable into the baseline SSTable of the target partition if the number of data to be deleted recorded in the incremental SSTable reaches a first quantity threshold, so as to delete the data to be deleted.

[0007] According to a third aspect of one or more embodiments of this specification, an electronic device is proposed, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the method of the first aspect by running the executable instructions.

[0008] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method of the first aspect are implemented.

[0009] The method provided in this specification can be applied to a database based on a log-structured merge tree storage structure. By obtaining the number of data to be deleted recorded in the incremental SSTable of the target partition, and then when the number of data to be deleted recorded in the incremental SSTable reaches a first threshold, the incremental SSTable is merged into the baseline SSTable of the target partition to delete the data to be deleted. Therefore, this specification can delete the data to be deleted in the target partition in a timely manner, thereby optimizing the query performance of the database and reducing the unnecessary disk space occupied by the data to be deleted. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG1 is a schematic diagram of a database structure based on an LSM storage structure provided by an exemplary embodiment.

[0011] FIG2 is a flow chart of a data processing method provided by an exemplary embodiment.

[0012] FIG3 is a schematic structural diagram of a device provided by an exemplary embodiment.

[0013] FIG4 is a schematic structural diagram of a data processing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0014] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0015] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0016] In a database based on a log-structured merge tree storage architecture, batch sequential writes to disk have much higher performance than random writes. Therefore, in order to improve the database's write performance, data deletion is actually achieved through writes.

[0017] For example, upon receiving a data deletion instruction, the database will append a tombstone marker to the data to be deleted. During a query, if the most recent data entry corresponding to the same key value is found to have a tombstone marker, it indicates that the data has been deleted. The term "data to be deleted" in this specification refers to data with tombstone markers.

[0018] However, this approach increases disk usage and sacrifices read performance. Specifically, because data deletion is accomplished by appending a tombstone marker to the data, deleted data still physically exists in the database, taking up additional disk space. Furthermore, because data queries require traversing the entire database, query speeds can be significantly slow when the data volume is very large. Furthermore, since deleted data still physically exists in the database, the amount of data traversed during the query increases, significantly impacting query speed.

[0019] In view of this, the embodiment of this specification obtains the number of data to be deleted recorded in the incremental SSTable of the target partition, and when the number of data to be deleted recorded in the incremental SSTable reaches a first threshold, the incremental SSTable is merged into the baseline SSTable of the target partition to delete the data to be deleted. As a result, the embodiment of this specification can delete the data to be deleted in the target partition in a timely manner, thereby optimizing the query performance of the database and reducing the unnecessary disk space occupied by the data to be deleted.

[0020] For ease of understanding, the application scenario of the embodiment of this specification will be first introduced with reference to FIG1 .

[0021] The solution provided in this specification is applied to a database based on a Log Structured Merge Tree (LSM Tree) storage structure.

[0022] Figure 1 is a schematic diagram of a database structure based on a log-structured merge tree storage structure provided by an exemplary embodiment. As shown in Figure 1, the log-structured merge tree may include a memory table (Memtable) and a sorted string table (SSTable).

[0023] Among them, Memtable is an in-memory data structure in a log-structured merge tree. Since Memtable is stored in memory, data read and write operations in Memtable can be performed efficiently. When a write operation occurs in the database, the data to be written is first written to Memtable to maintain fast write performance. Among them, deletion can also be understood as writing. Deleting data essentially writes a tombstone mark in Memtable, rather than actually deleting it. Due to limited memory capacity, when the size of Memtable reaches a certain threshold, the data stored in Memtable will be dumped to SSTable on the hard disk to free up storage space in memory for new data writing.

[0024] SSTable is a hard disk data structure in a log-structured merge tree, used to persist data. Exemplarily, SSTable can be divided into incremental SSTable and baseline SSTable. Exemplarily, when the data in Memtable is dumped to the hard disk, the SSTable to which these data are written is the incremental SSTable. When the incremental SSTables obtained from multiple dumps accumulate to a certain number or size, they can be merged with the baseline SSTable. In the merging process, data with tombstone marks will be recycled, that is, the data with tombstone marks will be actually cleared from the disk, thereby reducing disk space usage and optimizing read performance.

[0025] It should be understood that, although not shown in Figure 1, the number of incremental SSTables can be one or more. Multiple incremental SSTables can also form a multi-level storage structure. For example, the incremental SSTable can be divided into a first-level incremental SSTable and a second-level incremental SSTable. The data in multiple first-level incremental SSTables can be merged into a second-level incremental SSTable, and the data in multiple second-level incremental SSTables can be merged into a baseline SSTable. This embodiment of the present specification does not limit this.

[0026] Next, the exemplary embodiments of this specification will be described in detail in conjunction with the above application scenarios.

[0027] First, the embodiments of this specification provide a data processing method, which can be executed by any electronic device.

[0028] FIG2 is a flow chart of a data processing method provided by an exemplary embodiment. The data processing method is applied to a database based on a log structure merged with a tree storage structure.

[0029] As shown in FIG2 , the data processing method provided in the embodiment of this specification includes the following steps.

[0030] S201, obtain the number of data to be deleted recorded in the incremental ordered string table SSTable of the target partition.

[0031] It should be noted that the target partition can be any partition in the database. For example, the number of target partitions can be one or more. In the case of multiple target partitions, the method shown in S201 to S202 can be referred to, and each target partition can be processed separately, thereby timely deleting the data to be deleted in each target partition.

[0032] In some embodiments, the target partition may be a partition in the database that has higher requirements for data query performance, or a partition that requires frequent data queries. Thus, by timely deleting the rows to be deleted in the partition, the range of data that needs to be traversed during data query can be narrowed, thereby improving query performance.

[0033] It should be noted that incremental SSTables can be generated from Memtable dumps. In distributed database systems, data is typically divided into multiple partitions, each storing a portion of the data. Each partition has a corresponding Memtable, which stores recent data changes (including data deletions) in memory.

[0034] When the size of the target partition's Memtable exceeds a certain threshold, the data in the Memtable can be dumped into the incremental SSTable to free up memory, and a new blank Memtable can be created to receive new data change operations.

[0035] In some application scenarios, since the machine running the database has large memory, it is difficult for the Memtable to trigger a dump through the size threshold. This may cause a large amount of data to be deleted to be recorded in the Memtable in the form of tombstone markers, and the incremental SSTable or baseline SSTable cannot be aware of it, resulting in the data to be deleted not being deleted in time.

[0036] Therefore, the trigger conditions for Memtable dump can be configured to dump the Memtable that records a large amount of data to be deleted into incremental SSTable in a timely manner, and then complete the deletion of the data to be deleted by merging the incremental SSTable into the baseline SSTable.

[0037] For example, the number of data to be deleted recorded in the Memtable of the target partition may be obtained. If the number of data to be deleted recorded in the Memtable reaches a second threshold, the Memtable is dumped as an incremental SSTable.

[0038] Specifically, by scanning the data deletion statements (Delete) stored in the Memtable, the number of data to be deleted recorded in the Memtable can be determined. When the number of data to be deleted recorded in the Memtable is too large, it is believed that the data to be deleted may affect the target partition's external data query capabilities. Therefore, the Memtable can be dumped into an incremental SSTable in a timely manner so that the incremental SSTable can be merged with the baseline SSTable to perform actual deletion processing on the data to be deleted in a timely manner.

[0039] In some application scenarios, a large amount of data to be deleted may be generated in a short period of time in the target partition. For example, if a data delete statement is used to continuously delete data within a specified range in the target partition, using the number of pending data as the trigger for a dump may trigger multiple dumps in a short period of time, resulting in poor database performance.

[0040] For example, assuming the second threshold is 10,000 rows, that is, when the number of rows to be deleted in the Memtable reaches 10,000, the Memtable is dumped as an incremental SSTable. In this case, if 60,000 rows of data are deleted from the target partition in succession, six dumps will be triggered in succession, thereby occupying system resources in the database for a long time and causing database performance degradation.

[0041] In view of this, on the basis of using the amount of data to be deleted as the dump triggering condition, the time interval between two dumps can be limited to avoid frequent dump triggering in the target partition.

[0042] For example, the creation time of the Memtable may be obtained. If the time interval between the creation time and the current time is greater than a third preset time length, the Memtable is dumped as an incremental SSTable.

[0043] The creation time of the Memtable refers to the creation time of the currently active Memtable, that is, the creation time of the Memtable currently used to record data change operations. It is understood that the Memtable consists of two parts: the active MemTable and the frozen MemTable. When the above dump trigger conditions are met, the freeze (pre-dump action) will be automatically triggered, and the original active MemTable will become the frozen MemTable, and a new active MemTable will be generated. The dump will then be automatically scheduled, and the memory occupied by the frozen MemTable will be released after the dump is completed.

[0044] Therefore, by setting the third preset time length, it means that the active Memtable can trigger the dump process only after the third preset time length is created, thereby reducing the impact of frequent dumps on database performance to a certain extent.

[0045] It is worth noting that the above-mentioned method of triggering a dump based on the amount of data to be deleted, or the method of triggering a dump based on the amount of data to be deleted combined with a third preset duration, can be understood as one of the dump triggering methods configured in the target partition, and is not necessarily the only dump triggering method in the target partition. Other dump triggering methods can still be configured in the target partition, for example, triggering a dump when the size of the Memtable exceeds a certain threshold, triggering a dump when the length of time the Memtable has existed reaches a certain threshold, etc. Of course, users can also manually trigger a Memtable dump.

[0046] For example, even though the amount of data to be deleted recorded in the Memtable does not reach the second quantity threshold, the Memtable can still be dumped as an incremental SSTable if the size of the Memtable itself exceeds the preset size threshold.

[0047] That is to say, the incremental SSTable in the embodiments of this specification is not necessarily formed by the above-mentioned Memtable dump triggered by the amount of data to be deleted, but may also be formed by a Memtable dump triggered by other methods. The embodiments of this specification do not limit this.

[0048] Furthermore, it is understood that the number of incremental SSTables in the target partition may be one or more. For example, when multiple dumps are triggered in the target partition, each dump will generate a corresponding incremental SSTable. If, after multiple dumps are triggered, the resulting incremental SSTables are still not merged into the baseline SSTable, multiple incremental SSTables will exist in the target partition simultaneously.

[0049] That is to say, when the embodiments of this specification obtain the number of data to be deleted recorded in the incremental SSTable of the target partition, what is obtained is the number of data to be deleted recorded in all incremental SSTables in the target partition.

[0050] In some embodiments, the number of data to be deleted recorded by the incremental SSTable can be obtained by reading the data to be deleted stored in the Memtable during the dump process. It can be understood that regardless of the number of incremental SSTables, the incremental SSTables are obtained by dumping the Memtable, and during the dump process, the incremental SSTables can read all the data stored in the Memtable. Therefore, by counting the number of data to be deleted stored in the Memtable, the number of data to be deleted recorded by the incremental SSTable can be obtained.

[0051] S202: If the number of data to be deleted recorded in the incremental SSTable reaches a first quantity threshold, the incremental SSTable is merged into the baseline SSTable of the target partition to delete the data to be deleted.

[0052] It should be noted that when the number of dumps exceeds a certain threshold, or during a period of low database access, the database system will merge the incremental SSTables obtained from the dump into the baseline SSTable. In the new baseline SSTable obtained after the merger, the data to be deleted stored in the incremental SSTable will be deleted, thereby freeing up disk space and optimizing query performance.

[0053] It is understandable that the data to be deleted may exist in the incremental SSTable or the baseline SSTable. For example, when both the insert record and the delete record for the same data to be deleted exist in the incremental SSTable, deleting the data to be deleted means deleting the data to be deleted in the incremental SSTable. For data originally existing in the baseline SSTable, if a tombstone is found in the latest version of the data recorded in the incremental SSTable during the merge process, the data can be deleted from the baseline SSTable.

[0054] In some embodiments, the first quantity threshold may be the same as the second quantity threshold, or may be less than the second quantity threshold. When the number of data to be deleted recorded in the Memtable reaches the second quantity threshold, the Memtable will be dumped as an incremental SSTable. In this case, since the first quantity threshold is less than or equal to the second quantity threshold, when the number of data to be deleted recorded in the Memtable reaches the second quantity threshold, the number of data to be deleted recorded in the incremental SSTable obtained after the dump will also reach the first quantity threshold. Therefore, this method can quickly trigger the merging process of the incremental SSTable and the baseline SSTable after the Memtable is dumped, thereby improving the efficiency of deleting the data to be deleted.

[0055] In some embodiments, the first quantity threshold can also be greater than the second quantity threshold. In this case, even if the number of pending data recorded in the Memtable reaches the second quantity threshold, it does not mean that the number of pending data recorded in the incremental SSTable has reached the first quantity threshold. Because the merging process consumes far more database system resources than the dump process, performing merge operations at the frequency of dumps may cause a large amount of system resources to be occupied for a long time. Because the accumulation of pending data and the merging process of incremental SSTables may both affect the performance of the database system, by setting the first quantity threshold greater than the second quantity threshold, the impact of the two on database performance can be dynamically balanced. For example, when the accumulated amount of pending data reaches the first quantity threshold, it is considered that the adverse impact of the accumulated pending data on the database system is greater than the adverse impact of the merge process on the database system. At this time, the merge operation is performed first to delete the pending data. When the accumulated amount of pending data is less than the first quantity threshold, it is considered that the adverse impact of the accumulated pending data on the database system is less than the adverse impact of the merge process on the database system. In this case, there is no need to perform a merge operation. The pending data can be deleted by performing a merge operation when the accumulated amount of pending data reaches the first quantity threshold.

[0056] In some application scenarios, there may be multiple replicas of the target partition, and the multiple replicas include the master replica (Leader) and slave replicas (Follower) of the target partition. Among them, the master replica is responsible for receiving and processing write operations for the data in the partition, and synchronizing the written data to the slave replica to ensure the consistency of the data in the master and slave replicas. For example, the master replica will record the order and content of the write operations through a maintenance log (clog). The slave replica will regularly synchronize to the log from the master replica and replay the log so that the write operations recorded therein can be applied one by one to its own data copy, thereby ensuring the consistency of the data between the master and slave replicas. For data read operations, the database system can send read requests directly to the master replica, or send read requests to the slave replica for processing, thereby performing load balancing between the master and slave replicas and improving the read performance and throughput of the database.

[0057] In some embodiments, for each replica of the target partition, in the process of merging the incremental SSTable of the replica into the baseline SSTable, it is also necessary to consider the consistency of the data in each replica before and after the merge, and the master replica issues a unified merge instruction to the slave replica. Therefore, S202 may include: if the number of data to be deleted recorded in the incremental SSTable of the master replica reaches a first quantity threshold, then merge the incremental SSTable of each replica in the target partition into the corresponding baseline SSTable, and make the baseline SSTables of each replica after the merge consistent.

[0058] Understandably, because the merge process involves multiple replicas, the consistency of data in each replica before and after the merge must be considered, and consistency checks must be performed on both the master and slave replicas after the merge is complete. Therefore, the merge process involving multiple replicas is subject to numerous limitations and a lengthy triggering process. For example, if some slave replicas have not synchronized their logs to the master replica, or if log playback has not been completed, the merge process cannot be initiated. Consequently, the merge process involving multiple replicas is time-consuming and consumes significant resources in the database system.

[0059] Exemplarily, the time interval between two adjacent merging of the incremental SSTable of each replica in the target partition into the corresponding baseline SSTable is greater than the first preset duration.

[0060] That is to say, each time the merge of the incremental SSTable and the baseline SSTable of multiple replicas is triggered, the next merge is allowed to be triggered only after the first preset time period has passed, thereby avoiding the impact of frequent triggering of the merge operation of multiple replicas on database performance.

[0061] It is worth noting that the merging of the incremental SSTable involving multiple copies and the baseline SSTable triggered by the first quantity threshold is a triggering method that can improve the efficiency of deleting the data to be deleted, but it is not necessarily the only merge triggering method in the target partition. Other merge triggering methods can still be configured in the target partition, for example, triggering according to the size or number of incremental SSTables, triggering at a fixed time, manual triggering by the user, and so on. When these merge triggering conditions are met, the merging of the incremental SSTable involving multiple copies and the baseline SSTable will be triggered, and the consistency of the data in each copy before and after the merge needs to be ensured.

[0062] That is to say, when calculating the time interval between two adjacent merge processes, the previous merge process can be a merge of incremental SSTable and baseline SSTable involving multiple copies triggered by the first quantity threshold, or it can be a merge involving multiple copies triggered by other methods. The embodiments of this specification do not limit this.

[0063] In some embodiments, the merging process of each replica can be performed independently without considering the consistency between the replicas, thereby improving the merging efficiency of the incremental SSTable and the baseline SSTable, and further improving the data query efficiency. In this case, S202 may include: for each replica of the target partition, perform the following processing respectively: If the number of data to be deleted recorded in the incremental SSTable of the replica reaches a first quantity threshold, the baseline SSTable of the replica is redundantly stored. The incremental SSTable of the replica is merged into the redundantly stored baseline SSTable. The merged redundantly stored baseline SSTable is used to process data query requests for the replica.

[0064] In other words, the redundantly stored baseline SSTable in each replica can be merged independently without considering consistency with other replicas. Therefore, compared to the merging process involving multiple replicas initiated by the primary replica, the call link for merging the incremental SSTable with the redundantly stored baseline SSTable is shorter and more timely, and can complete the merging process at a lower cost and provide external data reading capabilities, thereby improving the query performance of the database.

[0065] When the next merge process that ensures consistency among multiple replicas is completed, the original baseline SSTable in each replica has been merged with the incremental SSTable, and the newer version of the data is recorded. Therefore, the redundantly stored baseline SSTable can be deleted, freeing up the disk space occupied by the redundantly stored baseline SSTable.

[0066] In some embodiments, the two merge strategies described above may be used in combination to achieve a certain degree of balance between performance consumption and disk usage. In this case, S202 may include: if the time interval between the last execution of the first merge strategy and the current time is greater than a second preset time duration, then executing the first merge strategy; otherwise, executing the second merge strategy.

[0067] The first merge strategy includes: if the number of data to be deleted recorded in the incremental SSTable of the primary replica reaches a first threshold, the incremental SSTable of each replica in the target partition is merged into the corresponding baseline SSTable. The baseline SSTables of each replica after the merger are consistent.

[0068] The second merge strategy includes: for each replica of the target partition, performing the following processing respectively: if the number of data to be deleted recorded in the incremental SSTable of the replica reaches a first quantity threshold, then redundantly storing the baseline SSTable of the replica. Merging the incremental SSTable of the replica into the redundantly stored baseline SSTable, wherein the merged redundantly stored baseline SSTable is used to process data query requests for the replica.

[0069] That is to say, the first merge strategy is the above-mentioned merge strategy initiated by the master replica involving multi-replica consistency, while the second merge strategy is the above-mentioned strategy independently performed by each replica to merge the incremental SSTable with the redundantly stored baseline SSTable. In the case where the execution frequency of the first merge strategy is low, the first merge strategy can be executed first to save the additional disk space occupied by redundant storage. In the case where the execution frequency of the first merge strategy is high, the second merge strategy can be executed, thereby improving the performance of the database by occupying a certain amount of additional disk space.

[0070] It is understandable that in actual applications, users can choose any of the above merge strategies according to their own needs. For example, when users have high requirements for database query performance, they can adopt the above merge strategy involving redundant storage SSTables. When users are more concerned about disk usage, they can adopt the above merge strategy initiated by the master copy to ensure multi-copy consistency. When users adopt a merge strategy that combines the two, they can achieve a certain degree of balance between data query performance and disk usage.

[0071] Therefore, this specification can delete the data to be deleted in the target partition in a timely manner, thereby optimizing the query performance of the database and reducing unnecessary disk space occupation by the data to be deleted.

[0072] FIG3 is a schematic diagram of the structure of a device provided by an exemplary embodiment. Referring to FIG3 , at the hardware level, the device includes a processor 302, an internal bus 304, a network interface 306, a memory 308, and a non-volatile memory 310, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 302 reading the corresponding computer program from the non-volatile memory 310 into the memory 308 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0073] Please refer to Figure 4, which provides a data processing device 400 that can be applied to the device shown in Figure 3 to implement the technical solution of this specification. Exemplarily, the data processing device 400 can be applied to a database based on a log-structured merge tree storage structure. Specifically, the data processing device 400 may include: an acquisition module 401 for acquiring the number of data to be deleted recorded in the incremental ordered string table SSTable of the target partition; a merging module 402 for merging the incremental SSTable into the baseline SSTable of the target partition if the number of data to be deleted recorded in the incremental SSTable reaches a first quantity threshold, so as to delete the data to be deleted.

[0074] In some embodiments, the target partition has multiple replicas. The merge module 402 is specifically used to perform the following processing for each replica of the target partition: if the number of data to be deleted recorded in the incremental SSTable of the replica reaches a first quantity threshold, the baseline SSTable of the replica is redundantly stored; the incremental SSTable of the replica is merged into the redundantly stored baseline SSTable, wherein the merged redundantly stored baseline SSTable is used to process data query requests for the replica.

[0075] In some embodiments, the target partition has multiple replicas, including a master replica and a slave replica. The merge module 402 is specifically configured to merge the incremental SSTable of each replica in the target partition into the corresponding baseline SSTable if the number of data to be deleted recorded in the incremental SSTable of the master replica reaches a first quantity threshold; wherein the baseline SSTables of each replica after the merge are consistent.

[0076] In some embodiments, the time interval between two adjacent merging of the incremental SSTable of each replica in the target partition into the corresponding baseline SSTable is greater than the first preset duration.

[0077] In some embodiments, the target partition has multiple replicas, including a master replica and a slave replica. The merge module 402 is specifically configured to execute the first merge strategy if the time interval between the last execution of the first merge strategy and the current time is greater than a second preset time interval, otherwise, execute the second merge strategy.

[0078] Among them, the first merge strategy includes: if the number of data to be deleted recorded in the incremental SSTable of the primary replica reaches a first quantity threshold, the incremental SSTable of each replica in the target partition is merged into the corresponding baseline SSTable; wherein, the baseline SSTables of each replica after the merger are consistent.

[0079] The second merge strategy includes: for each replica of the target partition, performing the following processing respectively: if the number of data to be deleted recorded in the incremental SSTable of the replica reaches the first quantity threshold, then redundantly storing the baseline SSTable of the replica; merging the incremental SSTable of the replica into the redundantly stored baseline SSTable, wherein the merged redundantly stored baseline SSTable is used to process data query requests for the replica.

[0080] In some embodiments, the acquisition module 401 is also used to obtain the number of data to be deleted recorded in the memory table Memtable of the target partition; if the number of data to be deleted recorded in the Memtable reaches a second quantity threshold, the Memtable is dumped as an incremental SSTable.

[0081] In some embodiments, the acquisition module 401 is further used to obtain the creation time of the Memtable; if the time interval between the creation time and the current time is greater than a third preset duration, the Memtable is dumped as an incremental ordered string table SSTable.

[0082] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0083] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0084] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0085] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0086] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0087] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0088] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0089] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0090] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A data processing method, applied to a database based on a log-structured merge tree storage structure, the method comprising: Get the number of data to be deleted recorded in the incremental ordered string table SSTable of the target partition; If the number of data to be deleted recorded in the incremental SSTable reaches a first quantity threshold, the incremental SSTable is merged into the baseline SSTable of the target partition to delete the data to be deleted.

2. The method according to claim 1, wherein the target partition has multiple replicas, and if the amount of data to be deleted recorded in the incremental SSTable reaches a first threshold, merging the incremental SSTable into the baseline SSTable of the target partition, comprising: For each replica of the target partition, perform the following processing: If the number of data to be deleted recorded in the incremental SSTable of the replica reaches a first quantity threshold, the baseline SSTable of the replica is redundantly stored; The incremental SSTable of the replica is merged into the redundantly stored baseline SSTable, where the merged redundantly stored baseline SSTable is used to process data query requests for the replica.

3. The method according to claim 1, wherein the target partition has multiple replicas, the multiple replicas including a master replica and a slave replica, and if the amount of data to be deleted recorded in the incremental SSTable reaches a first threshold, merging the incremental SSTable into the baseline SSTable of the target partition, comprising: If the number of data to be deleted recorded in the incremental SSTable of the primary replica reaches a first quantity threshold, the incremental SSTable of each replica in the target partition is merged into the corresponding baseline SSTable; Among them, the baseline SSTables of each merged replica are consistent.

4. According to the method of claim 3, the time interval between two adjacent merging of the incremental SSTable of each replica in the target partition into the corresponding baseline SSTable is greater than the first preset duration.

5. The method according to claim 1, wherein the target partition has multiple replicas, the multiple replicas including a master replica and a slave replica, and if the amount of data to be deleted recorded in the incremental SSTable reaches a first threshold, merging the incremental SSTable into the baseline SSTable of the target partition, comprising: If the time interval between the last execution of the first merging strategy and the current time is greater than the second preset time length, the first merging strategy is executed; otherwise, the second merging strategy is executed; The first merging strategy includes: If the number of data to be deleted recorded in the incremental SSTable of the primary replica reaches a first quantity threshold, the incremental SSTable of each replica in the target partition is merged into the corresponding baseline SSTable; wherein the baseline SSTables of each replica after the merger are consistent; The second merging strategy includes: For each replica of the target partition, perform the following processing: If the number of data to be deleted recorded in the incremental SSTable of the replica reaches a first quantity threshold, the baseline SSTable of the replica is redundantly stored; The incremental SSTable of the replica is merged into the redundantly stored baseline SSTable, where the merged redundantly stored baseline SSTable is used to process data query requests for the replica.

6. The method according to claim 1, further comprising: Obtain the number of data to be deleted recorded in the memory table Memtable of the target partition; If the number of data to be deleted recorded in the Memtable reaches a second quantity threshold, the Memtable is dumped as an incremental SSTable.

7. The method according to claim 6, wherein dumping the memtable into an incremental SSTable comprises: Get the creation time of the Memtable; If the time interval between the creation time and the current time is greater than the third preset duration, the Memtable is dumped as an incremental ordered string table SSTable.

8. A data processing device, applied to a database based on a log-structured merge tree storage structure, the device comprising: The acquisition module is used to obtain the number of data to be deleted recorded in the incremental ordered string table SSTable of the target partition; A merging module is used to merge the incremental SSTable into the baseline SSTable of the target partition if the number of data to be deleted recorded in the incremental SSTable reaches a first quantity threshold, so as to delete the data to be deleted.

9. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images