Data processing method, computing device and storage medium

Abstract

A data processing method, a computing device and a storage medium. The data processing method is applied to a storage engine of a database, the storage engine using a row number to record a write-in version number and a deletion version number of each row, and the storage engine comprising an active memory table and an inactive memory table. The method comprises: writing deletion mark data into an active memory table, the deletion mark data comprising a row number and a deletion version number of deleted data, so as to determine, in the case of responding to a data query operation, data visibility of the corresponding row number on the basis of the deletion mark data; if the remaining space of the active memory table is less than or equal to a preset space threshold value, converting the active memory table into an inactive memory table; and in response to a disk flushing operation, writing the deletion mark data in the inactive memory table into a persistent storage device. Thus, the problem that deletion mark data occupies a large amount of memory resources is avoided.

Description

Data processing methods, computing devices, and storage media

[0001] This disclosure claims priority to Chinese Patent Application No. 202411183623.7, filed with the China Patent Office on August 26, 2024, entitled “Data Processing Method, Computing Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of database technology, and in particular to a data processing method. One or more embodiments of this disclosure also relate to a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0003] With the continuous development of computer technology and data processing technology, in the process of performing data processing operations such as data updating and data deletion, the use of concurrent data processing operations can lead to data inconsistency.

[0004] To address the aforementioned issues, existing technologies assign a corresponding version number to each data processing operation to ensure data consistency during processing. However, this version number data consumes a significant amount of memory resources, severely impacting the normal use of memory. Therefore, how to prevent version number data from consuming excessive memory resources has become an urgent problem to solve. Summary of the Invention

[0005] In view of the above, embodiments of this disclosure provide a data processing method. One or more embodiments of this disclosure also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the present disclosure, a data processing method is provided, applied to a storage engine of a database. The storage engine uses row numbers to record the write version number and delete version number of each row. The storage engine includes an active memory table and an inactive memory table, comprising:

[0007] The deletion marker data is written to the active memory table, wherein the deletion marker data includes the row number of the deleted data and the deletion version number, so that in response to a data query operation, the data visibility of the corresponding row number can be determined based on the deletion marker data;

[0008] If the remaining space of the active memory table is less than or equal to a preset space threshold, then the active memory table is converted into an inactive memory table.

[0009] In response to the disk write operation, the deletion marker data in the inactive memory table is written to the persistent storage device.

[0010] According to a second aspect of the present disclosure, a data processing apparatus is provided, which is applied to a storage engine of a database. The storage engine uses row numbers to record the write version number and delete version number of each row. The storage engine includes an active memory table and an inactive memory table, comprising:

[0011] The data writing module is configured to write deletion marker data to the active memory table, wherein the deletion marker data includes the row number of the deleted data and the deletion version number, so that in response to a data query operation, the data visibility of the corresponding row number can be determined based on the deletion marker data;

[0012] The memory table conversion module is configured to convert the active memory table into an inactive memory table if the remaining space of the active memory table is less than or equal to a preset space threshold.

[0013] The memory table write module is configured to write deletion marker data from the inactive memory table to a persistent storage device in response to a disk write operation.

[0014] According to a third aspect of the present disclosure, a computing device is provided, comprising:

[0015] Memory and processor;

[0016] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0017] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0019] In the data processing method provided in one or more embodiments of this disclosure, during the processing of deletion marker data, newly generated deletion marker data is written into the active memory table in the data merging tree. When it is determined that the active memory table is full, the active memory table is converted into an inactive memory table. Furthermore, in response to the disk write operation, the inactive memory table is written into a persistent storage device to generate a new file. By writing the deletion marker data in memory into the persistent storage device, the problem of deletion marker data occupying a large amount of memory resources is avoided, and memory resources can be used normally. Attached Figure Description

[0020] Figure 1 is a schematic diagram of a data storage model provided in an embodiment of this disclosure;

[0021] Figure 2 is a schematic diagram of a mask component provided in an embodiment of this disclosure;

[0022] Figure 3 is a schematic diagram of a first scheme using pk tagging provided in an embodiment of this disclosure;

[0023] Figure 4 is a schematic diagram of a second scheme using pk tagging provided in an embodiment of this disclosure;

[0024] Figure 5 is a schematic diagram of a first bitmap marker scheme provided in an embodiment of this disclosure;

[0025] Figure 6 is a schematic diagram of a second bitmap labeling scheme provided in an embodiment of this disclosure;

[0026] Figure 7 is a schematic diagram of a third bitmap labeling scheme provided in an embodiment of this disclosure;

[0027] Figure 8 is a flowchart of a data processing method provided in an embodiment of this disclosure;

[0028] Figure 9 is a schematic diagram of a data merging tree in a data processing method provided in an embodiment of this disclosure;

[0029] Figure 10 is a schematic diagram of parallel scanning in a data processing method provided in an embodiment of this disclosure;

[0030] Figure 11 is a schematic diagram of multi-path parallel merging in a data processing method provided in an embodiment of this disclosure;

[0031] Figure 12 is a schematic diagram of multi-level data merging in a data processing method provided in an embodiment of this disclosure;

[0032] Figure 13 is a schematic diagram of a dual l sm-tree design in a data processing method provided in an embodiment of this disclosure;

[0033] Figure 14 is a flowchart of a data processing method provided in an embodiment of this disclosure;

[0034] Figure 15 is a structural block diagram of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0035] Numerous specific details are set forth in the following description to provide a full understanding of this disclosure. However, this disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this disclosure. Therefore, this disclosure is not limited to the specific implementations disclosed below.

[0036] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this disclosure, and similarly, second may also be referred to as first. Depending on the context, the word “if” as used herein may be interpreted as “when”, “in response to a determination”, or “when…”.

[0038] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0039] First, the terms and concepts involved in one or more embodiments of this disclosure will be explained.

[0040] HTAP: Hybrid Transactional / Analytical Processing.

[0041] mem: active memtable, the active memory table.

[0042] imm: immutable memtable, an inactive memory table.

[0043] rowid: row number.

[0044] seq: sequence, transaction sequence number.

[0045] compaction: to merge.

[0046] mask: a mask or cover.

[0047] lsm-tree: log-structed merge tree.

[0048] B+tree: a multi-way balanced tree.

[0049] FileStore: The disk-based data structure for the lsm-tree mask.

[0050] Checkpoint: A snapshot of the running system.

[0051] ACID: The four properties of database transactions.

[0052] copy-on-write: Copy on write.

[0053] Append-only is a property of computer data storage that allows new data to be appended to storage, but existing data remains immutable.

[0054] An append-only write model can be understood as a model where all data is written in an append-only manner, and modification of existing data is not supported. This model is commonly found in message queues and databases.

[0055] delta space: refers to the area used to store data changes or incremental data.

[0056] Stable space refers to the area where data is stored that is not easily changed or is updated infrequently.

[0057] SQL: Structured Query Language.

[0058] delta delete buffer: A cache used to store the deletion version number corresponding to the data change.

[0059] SQL Server: A database product.

[0060] With the continuous development of computer technology and data processing technology, during the process of performing data processing operations such as data update and data deletion, due to the concurrent execution of data processing operations, the problem of data inconsistency will occur. For example, the data processing method provided by one or more embodiments of the present disclosure can be applied to a database, which can be understood as a low-latency, high-performance HTAP-type database that realizes online transaction analysis integration. To achieve fast writing, the database kernel can adopt a heap table model. The append-only writing model of the heap table avoids the in-place update strategy with high cost read-write, thereby ensuring that updates can be quickly applied to the existing data. Specifically, refer to FIG. 1. FIG. 1 is a schematic diagram of a data storage model provided by an embodiment of the present disclosure. The data storage model shown in FIG. 1 can be the storage layer in the above database. This data storage model is a columnar storage model, as shown in FIG. 1. A relational table will be divided into multiple row groups (pack groups) according to a fixed number of rows. Each row group will be divided into multiple packs (data packets) according to columns. Each pack only stores a certain column and is physically stored continuously. In this way, efficient compression of the same data type on a single column can be achieved, and only the required columns need to be read during table scanning calculation, greatly reducing the I / O overhead.

[0061] Due to the heap table model of the database, in-place update cannot be performed during data update. Therefore, data update is achieved by appending new data after the existing data and writing the original data to a separate deletion mark. Also, due to the existence of concurrent update transactions in the database, to meet the ACID requirements of the transactions, both data writing and deletion are versioned. These version data (also called version numbers) are implemented through the mask component in FIG. 1. For example, as shown in FIG. 2 below, FIG. 2 is a schematic diagram of the mask component provided by an embodiment of the present disclosure. Based on FIG. 2, it can be seen that the mask component is logically two arrays (ins array, del array). Insertion and deletion both need to mark the version number (seq) of the transaction on the corresponding row. When reading (read), a read version number (read seq) is obtained from the system, and then it is respectively operated with the versions of the insert data and delete data corresponding rows. If read seq >= ins seq, the data in this row is visible; if read seq < del seq, the data in this row has not been deleted; where INF represents infinity.

[0062] The above calculations determine whether each row of data is visible. Since mask data has high performance requirements, each query must first scan the mask data and calculate the visibility of the corresponding rows. If the mask performance is insufficient, it will slow down all queries. To ensure high performance in mask calculations, the current database stores all mask data in memory. During execution, to prevent concurrent transactions from blocking each other, the following method can be used:

[0063] When a transaction writes a row, it writes the transaction version number to the `ins mask` of the row-level mask. When deleting a row, it writes the deletion transaction version number to the corresponding position in the `del mask` of that row. Updating a row involves deleting the old row and then writing the new data to the new row. Therefore, different transactions will not concurrently write to the mask of the same row.

[0064] The above describes the logical aspects. If the mask data structure is tightly coupled, such as a file-level bitmap or a complex treemap, from a computer architecture perspective, this type of data structure generally does not support concurrent writes. Even if different concurrent transactions write masks to different rows of this data structure, the structure itself does not support concurrent writes and can only be written serially. If a transaction writes a large amount of mask data, resulting in a long write time, and this data structure can only be written serially, other transactions will have to wait for this transaction to finish before they can write their own mask data, indirectly preventing transactions from executing concurrently.

[0065] However, the above method of storing all mask data in memory has the following two serious problems:

[0066] 1. The current mask is based on row groups, which consumes too much memory. Because it operates on a row group basis, even if only one row is deleted from a row group, the memory resources for the entire row group still need to be allocated. When the table is large, such as 10 billion rows of data, the required memory is approximately 75GB, which is excessive.

[0067] 2. High cost of data write-to-disk: When performing checkpoint operations, the mask needs to write all data to disk. When the data volume is large, this operation will be very time-consuming. In addition, the memory mask data needs to be completely rebuilt during startup, which takes a long time. In extreme cases, it can even take more than tens of minutes.

[0068] To address the aforementioned issues, and considering that columnar databases primarily use large data blocks for storage, in-situ updates are relatively costly. Therefore, this disclosure provides a mark-and-delete scheme. Mark-and-delete can be divided into pk markers and bitmap markers. The former is mainly used for index-organized tables, while the latter is used for heap tables.

[0069] PK (primary key) marking refers to primary key marking, which can be divided into LSM-tree merge and redo merge schemes. LSM-tree merge uses PK marking for deletion, and filters deleted rows during reading using PK merge. The redo merge scheme records the corresponding operations for the data blocks to be deleted, and applies the corresponding operations when the corresponding data blocks are read.

[0070] The PK (primary key) tagging scheme sacrifices data read performance because it requires a merge operation during reading. Furthermore, PK tagging typically uses incremental data for tagging, making it difficult to create secondary indexes on the incremental portion, thus limiting its application.

[0071] Specifically, the schemes using pk tags can include the following two schemes:

[0072] The first scheme, using PK (key-based) marking, marks records to be deleted using PK and performs a merge operation during reading, as shown in Figure 3. Figure 3 is a schematic diagram of the first PK-marking scheme provided in an embodiment of this disclosure. As shown in Figure 3, during data reading, updates are written to the delta space in the order they were written, and a global B+tree PK index is built on the delta store. During reading, for data (blocks) in the stable space, the B+tree is queried to confirm whether it is in the delta space. If it is in the delta space, it needs to be read, merged with the corresponding data in the stable space, and the latest data is calculated. It can be seen that this scheme requires reading both the delta space and the stable space simultaneously and performing a merge operation each time; this operation is CPU-bound, thus significantly impacting read performance.

[0073] The second scheme using pk marking records the update operations of each data block. Figure 4 is a schematic diagram of the second scheme using pk marking provided in an embodiment of this disclosure. As shown in Figure 4, this scheme divides the smallest unit of stored data into CUs. Updates are made by recording changes such as writes and deletions to the corresponding CUs. When the corresponding CU is read, if there is an update in the CU, the corresponding update is applied to the CU. This scheme has a low update cost, but it requires searching and merging updates when reading. However, it has two obvious drawbacks: (1) Since updates are made by recording changes, it is difficult to add secondary indexes; (2) CU updates will generate a lot of fragments, which require background defragmentation. Defragmentation also requires changing the corresponding update records, which is more complicated to implement.

[0074] Bitmap tagging is suitable for heap tables, where rows to be deleted are marked using independent components. However, the bitmap tagging scheme has significant drawbacks: First, updates are slow and require serialization; second, block-level updates still require serialization, necessitating the maintenance of multiple versions of the bitmap, resulting in severe write amplification; and finally, it consumes large amounts of memory and I / O resources.

[0075] Specifically, the schemes using bitmap marking can include the following three schemes:

[0076] The first bitmap labeling scheme uses a table-level bitmap to directly label the row numbers of the heap table. As shown in Figure 5 below, which is a schematic diagram of the first bitmap labeling scheme provided in an embodiment of this disclosure, for the heap table, deletion only requires setting the bitmap of the corresponding row to 0. During reading, the primary index is used to scan the bitmap to determine whether each row has been deleted. This scheme has high read efficiency, but it is not very friendly to concurrent transaction updates. Because concurrent transactions involve updating the global bitmap, and since the bitmap is table-level, concurrent updates need to be serialized through a global lock, resulting in low update efficiency. SQL Server improves its update performance through multiple means such as delta delete buffer, delayed application, and row memory assistance; however, the implementation of these schemes is complex and they are bound to the SQL Server kernel row memory, making it difficult to extend to other systems.

[0077] The second bitmap tagging scheme uses a block-level bitmap approach. See Figure 6 for details; Figure 6 is a schematic diagram of the second bitmap tagging scheme provided in an embodiment of this disclosure. By dividing the table into blocks, the contention for table-level bitmap updates is reduced. Specifically, the data written to disk for each table is grouped into multiple file blocks, each file block has a bitmap, and different bitmaps are stored using an LSM-tree. During updates, the filename is used as the key and the bitmap as the value, written to the LSM-tree. During reads, the corresponding bitmaps are merged to obtain the visibility of each row of data in each file. This scheme is faster than the table-level bitmap scheme in terms of write speed, but the cost of merging bitmaps during reads remains high; furthermore, single-row updates also require writing the entire row's bitmap, resulting in significant write amplification. ADB and Doris, to improve the performance of batch writes, have removed support for concurrent transactions and place the entire bitmap in memory, which is a simplification of the SQL Server approach.

[0078] The third bitmap labeling scheme uses row-level version numbers; see Figure 7 for details. Figure 7 is a schematic diagram of the third bitmap labeling scheme provided in an embodiment of this disclosure. Its core idea is to use two full-memory arrays to record the write version number and delete version number of each row. This scheme is simple and has very fast update and read speeds, but since the version arrays are full-memory, they consume a lot of memory resources, making it only suitable for scenarios with relatively small amounts of data. Furthermore, the entire array needs to be flushed to disk every time the system performs a checkpoint operation, consuming significant I / O resources.

[0079] In addition, this disclosure also provides a method using pure disk B+tree records.<rowid,seq> The proposed solution uses the inherent scanning method of B+trees to scan the mask. However, this solution has significant drawbacks. B+tree scanning can lead to severe cache congestion and a large amount of random disk I / O, resulting in poor performance.

[0080] Based on this, a data processing method is provided in this disclosure. One or more embodiments of this disclosure also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0081] Referring to Figure 8, which shows a flowchart of a data processing method according to an embodiment of the present disclosure, the data processing method is applied to the storage engine of a database. The storage engine uses row numbers to record the write version number and delete version number of each row. The storage engine includes an active memory table and an inactive memory table, and specifically includes the following steps.

[0082] Step 802: Write the deletion marker data into the active memory table, wherein the deletion marker data includes the row number of the deleted data and the deletion version number, so that in response to a data query operation, the data visibility of the corresponding row number can be determined based on the deletion marker data.

[0083] Specifically, this method can write newly generated deletion marker data into the active memory table corresponding to the data merging tree. The deletion marker data includes the row number and deletion version number of the deleted data. The deletion marker tree contains files in the persistent storage device and corresponding active and inactive memory tables in memory. This allows the data visibility of the corresponding row number to be determined based on the deletion marker data in response to data query operations.

[0084] The deletion marker data can be understood as data used to mark that the data has been deleted during the data deletion process. For example, the deletion marker data can be a version number, timestamp, etc.

[0085] In this context, persistent storage devices can be understood as devices used for persistent storage of data, such as disks, databases, etc.

[0086] The active memory table can be understood as a memory table used to store active deletion marker data; active deletion marker data can be understood as data whose generation time is less than or equal to a preset time threshold (e.g., 1 minute, 2 minutes); or, within a preset time range (e.g., 1 minute, 2 minutes), there are read operations for the deletion marker data; or, within a preset time range (e.g., 1 minute, 2 minutes), the number of read operations for the deletion marker data is greater than or equal to a preset number threshold (e.g., 10 times, 20 times).

[0087] The data merging tree can be understood as a tree-like storage structure used to store the deletion marker data. For example, the data merging tree can be understood as an LSM-tree structure. The data merging tree can be an LSM-tree mask, and the structure of the LSM-tree mask can be seen in Figure 9. Figure 9 is a schematic diagram of the data merging tree in a data processing method provided by an embodiment of this disclosure. The overall architecture of the LSM-tree mask is shown in Figure 9. It is an LSM-tree architecture, where mem represents the active memory table. There is only one in a single system to provide high-concurrency read and write operations. When mem is full, it is converted into a read-only imm memtable (imm in Figure 9) and waits to be flushed to disk. Multiple imm (i.e., inactive memory tables) may accumulate. These accumulated imm are connected together in the order of conversion to form a linked list. After imm is written to disk, multiple files (i.e., new files) will be created, i.e., FileStore in Figure 9.

[0088] Furthermore, these files are merged in descending order of size to form multiple layers, L0, L1, and L2 in Figure 9; the file's metadata is recorded in the meta tag; each file is divided into three information segments, where the meta tag records the file size and the position of each information segment; and the data pages record all deleted records.<rowid,seq> Each deleted record is stored in the form of <line number, deletion version number>. A Level-trie is a locator index that can pinpoint a specific page based on the line number, improving the efficiency of file seek operations. Files written to disk are divided into two types: normal layout and compressed layout. The normal layout includes the deletion version number, used to determine visibility for concurrent transactions. If all records in the file are visible, it is converted to a compressed layout, removing the sequence number (seq) and storing only the deletion number to save storage space.

[0089] The Level-trie can be understood as a hash table: page_id = hash(key).

[0090] In this approach, the compressed disk file is modified by removing the sequence number (seq) and retaining only the storage number. This means that the solution only records deletion marker data. In other words, the Mask component is logically an array that only records deleted records. For example, if a table has 100 million rows, using an array would require creating an array with 100 million elements in memory, which would be very memory-intensive. However, by directly recording the row number, such as for deleting only one row, only one row needs to be recorded, requiring only a few bytes of memory.

[0091] Since the database itself is a heap table structure, there is no in-situ update, which means that each row of each table will be deleted at most once. Therefore, the lsm-tree mask does not need to consider the complex multi-version merging problem, that is, each row of data has only one version.

[0092] In addition, lsm-tree mask provides two operation interfaces:

[0093] The `put(rowid, seq)` function adds a deletion marker to a row, where `rowid` is the row number and `seq` is the transaction version number of the deletion. Concurrent operations are supported.

[0094] `iterator = seek(rowid)` returns an iterator that starts at a position greater than or equal to `rowid`. The iterator can continuously scan for the next record by calling the `next` interface until the ending position is reached. Concurrent operations are supported.

[0095] Step 804: If the remaining space of the active memory table is less than or equal to a preset space threshold, then the active memory table is converted into an inactive memory table.

[0096] There can be one or more inactive memory tables.

[0097] The remaining space of the active memory table can be understood as the remaining capacity of the active memory table, such as the remaining number of rows or the remaining data storage space. The preset space threshold can be set according to the actual application scenario. For example, the preset space threshold can be a preset number of rows threshold (e.g., 10 rows) or a preset space threshold (e.g., 5% space threshold). When it is determined that the remaining space of the active memory table is less than or equal to the preset space threshold, the active memory table is considered full.

[0098] Continuing with the previous example, in the LSM-tree architecture, once the active memory table is full, it is converted into a read-only inactive memory table and awaits flushing to disk. Multiple imm may accumulate, and these accumulated imm are linked together in the order of conversion to form a linked list.

[0099] Step 806: In response to the disk write operation, write the deletion mark data in the inactive memory table to the persistent storage device.

[0100] Specifically, in response to the disk write operation, this solution writes the deletion marker data in the inactive memory table to the persistent storage device, and uses the data merging data in the persistent storage device to store the deletion marker data. Alternatively, the deletion marker data in the inactive memory table can be stored in the persistent storage device as a file.

[0101] It should be noted that the data merging tree may contain multiple files, which are used to store deletion marker data in the inactive memory table.

[0102] Following the previous example, after the imm file is written to disk, multiple files will be created. Among them, the data of the lsm-tree mask will be written to disk in two ways:

[0103] (1) When mem is full, it is converted to imm, and then the disk flush operation is called to flush it to the disk in the background and turn it into FileStore;

[0104] (2) When the number of FileSotre files on the disk reaches a certain amount, k files are selected from the n files and merged into one file according to a better strategy.

[0105] In fact, the former can be seen as merging an imm file with an empty FileStore. Therefore, the two methods of data persistence mentioned above are collectively referred to as merge operations. Compaction requires determining the K files to be merged each time, which involves calculating the optimal K and the selection of the K files. Classic LSM-trees typically employ leveled or tiered compaction strategies to address different application priorities for read and write operations. The read / write mode of LSM-tree masks is quite unique: 1. It needs to support fast write functionality; 2. It does not require point lookups, i.e., all queries are scans.

[0106] In one or more embodiments provided in this disclosure, the method can respond to a disk write operation by writing the deletion marker data in the inactive memory table to a persistent storage device to generate a new file, as detailed below:

[0107] The step of writing the deletion marker data in the inactive memory table to the persistent storage device includes:

[0108] A first preset number of files are selected from the files in the persistent storage device, and the first preset number of files are merged with the deletion mark data in the inactive memory table to generate a new file.

[0109] The first preset quantity can be set according to the actual application scenario. The first preset quantity can be zero, one or more, that is to say, this method can filter one file, multiple files or zero files.

[0110] Continuing with the previous example, the LSM-tree mask needs to be written as quickly as possible, therefore write amplification should be minimized, and a leveled strategy should not be used. Furthermore, the LSM-tree mask does not require an explicit data hierarchy mechanism like that of the LSM-tree. Therefore, a tiered strategy is also unsuitable. Thus, the compaction of the LSM-tree mask uses the following restricted merge-tree variant strategy, specifically:

[0111] LSM-tree masking does not explicitly distinguish the lifecycle of data; that is, during a query, it retrieves all data, performs a full scan, and merges the results. LSM-tree masking divides the data persisted to disk into multiple filestores (i.e., files), and each flush or compaction generates a new filestore. The maximum number of filestores is specified. Each flush and compaction evaluates the filestores and selects the best set (i.e., a first preset number of files) to merge into a new filestore (i.e., a new file). After merging, the old filestore is automatically deleted if no other thread references it.

[0112] In the above embodiments, by merging a first preset number of files with an inactive memory table, the inactive memory table is written to disk, thus avoiding the problem of deleting marked data occupying a large amount of memory resources.

[0113] In one or more embodiments provided in this disclosure, the step of filtering a first preset number of files from the files of the persistent storage device includes:

[0114] Based on file size, a first preset number of files are selected from the files in the persistent storage device.

[0115] Following the previous example, the principle for selecting filestores is to merge smaller filestores into larger ones as much as possible, that is, to minimize file fragmentation on disk. Since the key-value pair length in the lsm-tree mask is fixed, the number of key-value pairs in each filestore is used to represent the size of the file. Files are selected and scored according to the following rules.

[0116] The process of selecting files requires ensuring that at least two files exist. The preferred set of filestores is the objective of solving the following formula (1):

[0117] Where k represents the k files selected from n filestores, and store k This indicates the size of the selected k files. k can be a preset number.

[0118] There are two ways to select k files from n files:

[0119] 1. Starting from file i, select k files sequentially, so that the set of files is {i, i+1, i+2, ..., i+k}.

[0120] In other words, referring to Formula 1 above, starting from i=1, we can enumerate k consecutive files in sequence to select the file set. For example, if we want to select 3 out of 5 files, we can choose the permutations 1, 2, 3, 2, 3, 4, and 3, 4, 5, and then select the better one from the 3 permutations.

[0121] 2. Exhaustively search by permutation number, that is... Here, K can refer to k consecutive files.

[0122] Of these, scheme (2) can guarantee that the set of files to be elected is better, but when n is large, it is difficult to calculate the number of permutations, so it is suitable for cases where n is small. The lsm-tree mask can be configured as one of the above schemes according to requirements.

[0123] In the above embodiments, a first preset number of files are selected from the files in the persistent storage device based on file size, which facilitates subsequent file merging, thereby achieving the purpose of organizing the files in the persistent storage device and realizing efficient utilization of storage resources.

[0124] In one or more embodiments provided in this disclosure, the data processing method further includes:

[0125] A second preset number of files are selected from the files in the persistent storage device, and these files are merged to generate a new file. This new file is the one in the persistent storage device that stores the deletion marker data. In other words, this file is used to store the deletion marker data written to the persistent storage device.

[0126] The second preset quantity can be set according to the actual application scenario. The second preset quantity can be at least two, that is to say, this method can filter at least two files for merging to obtain a new file.

[0127] Continuing with the previous example, when the number of Filestores on the disk reaches a certain level, a better strategy is adopted to select k files (i.e., the second preset number) from n files and merge them into one file. Specifically, the above Flush operation can select k filestores. If k = 0, then a new file is directly generated based on the inactive memory table. If k is greater than or equal to 2, then multiple existing files need to be selected to participate in the merging to obtain a new file. As for the value of k, a better scheme needs to be calculated based on the merging algorithm.

[0128] For multiple filestores generated after data is written to disk, compaction can be used to process them, thereby generating a new filestore from multiple filestores. Each compaction evaluates multiple filestores and selects the better set of filestores (i.e., a second preset number of files) to merge into a new filestore (i.e., a new file). After the merger is completed, the old filestore is automatically deleted when no other thread references it, thereby achieving efficient utilization of storage resources.

[0129] In one or more embodiments provided in this disclosure, the data processing method further includes:

[0130] If the deletion marker data is visible in any file in the persistent storage device, then the deletion version number in that file is removed.

[0131] The phrase "deleted marked data is visible" can be understood as meaning that all data corresponding to deleted marked data is visible.

[0132] Continuing with the previous example, the disk file is divided into two types. If all records in the file are visible, it is converted into a compressed version, removing the seq version number and only storing the storage number to save storage space. Specifically, the Mask component is logically an array that only records deleted records. For example, if a table has 100 million records, using an array would require creating an array with 100 million elements in memory, which would consume a lot of memory. If the row number is recorded directly, such as when only one row is deleted, only one record needs to be recorded, which only occupies a few bytes of memory. Based on this, during the compression and storage process, the compressed disk file removes the seq version number and only stores the storage number, thereby saving storage resources.

[0133] In one or more embodiments provided in this disclosure, the data processing method further includes:

[0134] In response to a data query operation, determine the start row number and end row number corresponding to the data query operation;

[0135] The target object is determined from the data merging tree based on the start line number and the end line number, wherein the target object includes one or more of the active memory table, the inactive memory table, and the file;

[0136] Determine the initial data loading row number from the target object based on the starting row number;

[0137] The deletion marker data is loaded starting from the initial data loading row number, and the data visibility of the corresponding row number is determined based on the deletion marker data.

[0138] Here, the starting row number can be understood as the starting row number in the process of performing a data query operation on multiple row numbers; correspondingly, the ending row number can be understood as the ending row number in the process of performing a data query operation on multiple row numbers; the data query operation is performed within the range between the starting row number and the ending row number.

[0139] The initial data loading row number can be understood as the row number used to load the deleted data that was retrieved.

[0140] Using the previous example, the scan (data query operation) of the lsm-tree mask consists of two operations: seek (query) and scan (browse). When performing seek (key), it is necessary to seek (query) for each file written to disk to find the corresponding position that is greater than or equal to key. Here, key refers to the target position to be queried carried in the data query operation.

[0141] The key is the rowid. For example, to determine whether the row with rowid = 103 has been deleted, you would execute seek(103). The standard semantics of the seek(key) operation is to query records that are greater than or equal to the key. This is the standard interface semantics of the application service layer. For example, find(key) is to precisely query the record corresponding to the key. The semantics required by the upper layer of lsm-mask are the seek semantics.

[0142] The seek(key) operation for each file returns lower_bound(key), which is the first position greater than or equal to key. As shown in Figure 10, Figure 10 is a schematic diagram of parallel scanning in a data processing method provided in an embodiment of this disclosure. The data query operation can be seek(30), and the steps to perform the seek(30) operation are as follows:

[0143] First, the metadata of each file records the min-max index within that file, which is the minimum rowid (starting line number) and the maximum rowid (ending line number). During seek, files that do not need to be iterated are filtered out based on this index.

[0144] For example, if the rowid range of a file is [100, 500], seek(800) does not need to iterate over the file; however, due to the low-bound semantics of seek, seek(50) or seek(30) will need to query the file.

[0145] Secondly, for the filtered files (e.g., file 0, file 1, file 2), line number positioning is required based on seek(30); the filtered files can be understood as the target objects mentioned above.

[0146] Specifically, according to seek(30), for file0, it needs to be located at rowid=32 (i.e., the initial data loading line number), for file1, it needs to be located at rowid=30 (i.e., the initial data loading line number), and for file2, it needs to be located at rowid=55 (i.e., the initial data loading line number).

[0147] It should be noted that during iterative queries, the n files are sorted using multi-way merge sort.

[0148] Finally, starting from the position located in n files, obtain the version number (i.e., the deletion marker data), and determine the data visibility of the corresponding line number based on the version number.

[0149] It should be noted that the files written to disk are indexed by a level-trie, but the level-trie can only locate the page ID of the key. Therefore, it is necessary to load the page and perform a binary search to further determine the lower_bound(key). Thus, the seek operation for each hit file written to disk requires at most one I / O operation.

[0150] If the number of layers in the log-based mask file persisted to disk is too large, it may lead to significant overhead in seek operations. Therefore, the log-based mask should be merged in the background as much as possible to reduce the number of layers in the LSM-tree. Additionally, to speed up the scanning of each file and further reduce I / O during scanning, I / O prefetching is required for each file, for example, prefetching 64KB of data. This parameter can be dynamically configured.

[0151] Furthermore, to improve merging efficiency, the LSM-tree mask supports multi-path parallel merging. As shown in Figure 11, which is a schematic diagram of multi-path parallel merging in a data processing method provided in an embodiment of this disclosure, the scanning mask can create an LSM-tree iterator for each thread according to the configured concurrency, but different iterators choose different scanning start and end points. Logically, the mask data is divided according to the number of threads, and different threads scan different parts. Each thread scans the mask in units of logical pack groups, and each group contains a fixed number of rows, such as 64k (64*1024) rows. The LSM-tree mask calculates the starting rowid and ending rowid of the group according to the group id, and locates the start and end points of each layer of data in the LSM-tree mask that need to be read. Data location uses a full-memory level-trie index, which can be completed in O(1) time and does not require disk access. Since different scanning threads always scan different groups, the scanning mask can be parallelized to the maximum extent. From a data structure perspective, the data structures of files written to disk are static, thus eliminating concurrent read / write issues. Multi-threaded scan performance depends on disk bandwidth and the number of CPUs. The active memtable in memory requires concurrent read / write operations, but the read / write separation of the B+tree design described above allows for complete separation of data scanning and updating. Consequently, before reaching disk bandwidth, the scan performance in the LSM-tree mask exhibits a linear speedup with increasing CPU usage.

[0152] In one or more embodiments provided in this disclosure, determining the start row number and end row number corresponding to the data query operation includes:

[0153] Determine the row group corresponding to the data query operation, and set the starting row number and ending row number of the row group as the starting row number and ending row number of the data query operation. The row group is determined based on multiple row numbers.

[0154] Based on this, during the data query process, it is necessary to determine the row group corresponding to the data query operation in each file (i.e., the square that constitutes each file in Figure 10); and determine the starting and ending row numbers corresponding to the row group as the starting and ending row numbers corresponding to the data query operation, so as to facilitate the query.

[0155] In one or more embodiments provided in this disclosure, loading the deletion marker data starting from the initial data loading row number and determining the data visibility of the corresponding row number based on the deletion marker data includes:

[0156] A preset number of deletion marker data are loaded into memory from the initial data loading row number;

[0157] The deletion marker data in the memory is sorted to obtain ordered deletion marker data, and the data visibility of the corresponding row number is determined based on the ordered deletion marker data.

[0158] When the data loading conditions are met, the end line number of the previous data loading is used as the new initial data loading line number, triggering the step of loading a preset number of deletion marker data from the initial data loading line number into memory.

[0159] The preset data quantity can be set according to the actual application scenario. For example, the preset data quantity can be 64KB, 128KB, etc.; no specific limit is set here.

[0160] Specifically, in order to facilitate data processing and avoid putting excessive storage pressure on memory due to large amounts of data during the process of loading deletion marker data into memory, this method can adopt a batch writing approach to load the deletion marker data into memory; the specific method is as follows:

[0161] First, a preset number of deletion marker data is loaded from the initial data loading row number, and the deletion marker data is written into memory;

[0162] Secondly, the deletion marker data in memory is sorted to obtain ordered deletion marker data, and the data visibility of the corresponding row number is determined based on the ordered deletion marker data; for example, for an lsm-tree mask, the minimum value is taken out and its seq value is calculated to determine whether the current row has been deleted.

[0163] Finally, when it is determined that the data loading conditions are met, the end line number of the previous data loading is used as the new initial data loading line number, triggering the step of loading a preset number of deletion mark data from the initial data loading line number into memory, thereby realizing the batch loading of deletion mark data into memory and avoiding a large amount of data from putting a large storage pressure on memory.

[0164] In one or more embodiments provided in this disclosure, the memory includes a first cache and a second cache, and loading a preset number of deletion marker data from the initial data loading line number into the memory includes:

[0165] The deletion marker data of a first preset number is loaded from the initial data loading line number and stored in the first cache;

[0166] Load a second preset amount of deletion marker data into the second cache from the first cache;

[0167] Accordingly, sorting the deletion marker data in the memory to obtain ordered deletion marker data includes:

[0168] The deletion marker data is loaded from the second cache into the loser tree, and the loser tree outputs the ordered deletion marker data.

[0169] The first and second preset data quantities can be set according to the actual application scenario, and no specific restrictions are imposed here.

[0170] Continuing with the previous example, merging multi-level data is usually a merge sort process, and k-level data typically requires a k-way external sort. The implementation of lsm-tree mask uses a cached loser-tree to achieve k-way merging, and its merge performance is approximately 1.5 to 2 times that of a min-heap.

[0171] The implementation of multi-level merge is shown in Figure 12. Figure 12 is a schematic diagram of multi-level data merging in a data processing method provided by an embodiment of this disclosure. The data located in the Disk (memory) consists of k ordered files (i.e., deletion marker data of a first preset number of data); wherein the k ordered files are sorted in ascending order of rowid.

[0172] In DRAM, a two-level cache with a loser-tree for sorting is used. The L1 cache (the first cache) is a batch cache, used to load a batch of data from disk at a time, such as 64KB (the initial preset data size), to reduce random disk I / O operations. The L0 cache (the second cache) loads one key-value pair from the L1 cache at a time, primarily used to fill gaps after the loser-tree output.

[0173] The loser tree can be understood as an array of indices, with the top layer storing the index of the minimum value in the L0 cache. After outputting the minimum value of the loser tree, the loser tree will calculate the next minimum value in O(lnk) time.

[0174] The L0 cache is an array that records...<rowid,seq> The `loser-tree` is used to efficiently sort the array. The top node of the `loser-tree` records the index of the minimum value in the array, where the minimum value refers to the `rowid`. For example, if the top node of the `loser-tree` records 4, then the minimum value is the fourth element of the array. After finding the minimum value, it is removed from the `loser-tree`. Then, the next record is read from the file containing the fourth element of the array. The `loser-tree` is then adjusted, and after adjustment, the top node of the `loser-tree` will be the next minimum value.

[0175] Because the loser-tree and L0 cache contain relatively small amounts of data (maximum KB) and are stored contiguously, they offer the highest cache friendliness. After outputting the minimum value, its index indicates its position in the L0 cache, and then a copy of the data is loaded from the corresponding L1 cache. If the L1 cache is already full, the next batch of data is loaded from the corresponding file into the L1 cache.

[0176] For the lsm-tree mask, after extracting the minimum value, its seq value can be calculated to determine whether the current row has been deleted.

[0177] In one or more embodiments provided in this disclosure, the data processing method further includes:

[0178] In response to a checkpoint operation, incremental data in the data merge tree is copied to the checkpoint data tree, wherein the incremental data is deletion marker data between the last checkpoint operation and the currently executed checkpoint operation.

[0179] Continuing with the previous example, a checkpoint is a snapshot of the running data of the LSM-tree mask, which is then saved to disk. At startup, the data can be loaded from disk into memory to continue operation. The checkpoint mechanism should have the following characteristics: 1. Non-blocking: Checkpoints should not affect ongoing operations; this requires checkpoints to support non-blocking. 2. Transactional: Checkpoints should ensure the ACID properties of transactions.

[0180] Regarding the first requirement, the LSM-tree mask needs to support an asynchronous checkpointing mechanism; regarding the second requirement, checkpoints must not contain uncommitted data or invisible data. However, firstly, checkpointing is performed asynchronously in the background, so foreground operations still write to the LSM-tree mask during checkpointing. Secondly, according to the LSM-tree architecture paradigm, data already written to disk cannot be modified; if the written data contains future data, checkpointing operations will be impossible.

[0181] To address the above issues, the LSM-tree mask introduces a dual LSM-tree design with a copy-on-write mechanism. At runtime, the LSM-tree mask is divided into a checkpoint tree and a delta tree. The checkpoint tree is generated in the background, strictly guaranteeing the ACID semantics of IMCI transactions. It is controlled by the checkpoint sequence number and does not contain future data. The delta tree records the incremental portion of the LSM-tree mask, which may contain data greater than the latest checkpoint. Its primary function is to ensure that memtable data is flushed to disk promptly, reducing the memory footprint of the LSM-tree mask. The delta tree is automatically cleaned up upon restart and dynamically regenerated from the checkpoint tree at startup.

[0182] Based on this, when the lsm-tree mask starts, the checkpoint tree and delta tree share the same data, as shown in Figure 13. As the front end writes data, the two trees gradually split. The checkpoint tree generates data according to strict ACID transaction semantics, while the delta tree flushes the memtable to disk as quickly as possible to free up memory, based on the principle of saving limited memory.

[0183] During each checkpoint, the lsm-tree mask traverses the delta tree based on the input checkpoint sequence, filtering out outdated data and data that is no longer needed, and stores them in the checkpoint tree.

[0184] In this design, reads always access the delta tree. The checkpoint tree is primarily used to load data from a consistent transaction point at startup. At this point, the mask ensures that reads will not see future data or lose existing data. Secondly, checkpoints always record incremental data, significantly reducing disk I / O overhead compared to a full-scale disk flushing scheme using an entire in-memory array.

[0185] In one or more embodiments provided in this disclosure, copying incremental data from the data merging tree to the checkpoint data tree includes:

[0186] Filter out the target data from the incremental data to obtain the data to be copied. The target data includes the deletion marker data generated after this checkpoint operation and / or the deletion marker data that meets the data deletion conditions.

[0187] The data to be copied is copied to the checkpoint data tree.

[0188] In this context, data marked for deletion that meets the deletion criteria can be understood as data marked for deletion that is no longer needed. These deletion criteria can be set according to the actual application scenario. For example, if the data marked for deletion is future data (i.e., data from the future), then the data marked for deletion is determined to meet the deletion criteria. Alternatively, if the data marked for deletion is greater than the data at the latest checkpoint, then the data marked for deletion is determined to meet the deletion criteria.

[0189] Following the previous example, at each checkpoint, the lsm-tree mask traverses the delta tree based on the input checkpoint sequence, filtering out outdated data and data that is no longer needed, and stores them in the checkpoint tree.

[0190] In the data processing method provided in one or more embodiments of this disclosure, during the processing of deletion marker data, newly generated deletion marker data is written into the active memory table in the data merging tree. When it is determined that the active memory table is full, the active memory table is converted into an inactive memory table. Furthermore, in response to the disk write operation, the inactive memory table is written into a persistent storage device to generate a new file. By writing the deletion marker data in memory into the persistent storage device, the problem of deletion marker data occupying a large amount of memory resources is avoided, and memory resources can be used normally.

[0191] The following description, in conjunction with Figure 14, uses the application of the data processing method provided in this disclosure in an efficient data marking and deletion scenario for HTAP databases as an example to further illustrate the data processing method. Figure 14 shows a flowchart of the processing procedure of a data processing method provided in one embodiment of this disclosure. It should be noted that the data processing method provided in this disclosure is applied to a database and specifically includes the following steps.

[0192] Step 1402: When the database receives a data write request or a data delete request for a data table, it assigns a corresponding write version number to the data write request and a corresponding delete version number to the data delete request.

[0193] Specifically, the database used in this solution can adopt a columnar storage model. For any relational table, the columnar storage model will divide it into multiple row groups according to a fixed number of rows. Each row group will be divided into multiple packs according to columns. Each pack stores only one column and is physically stored contiguously.

[0194] Because a heap table model (i.e. columnar storage model) is used, it is difficult to perform efficient in-situ updates when updating data. Therefore, updates are achieved by appending new data to existing data and writing the original data to a separate deletion marker.

[0195] Meanwhile, due to concurrent update transactions in the database, to meet the ACID requirements of transactions, data write and delete operations based on data write or delete requests all have corresponding version numbers. These version data (i.e., version numbers) are implemented through the mask component in the database. Logically, this mask component consists of two arrays (i.e., the ins array and del array in Figure 2). Data insertion and data deletion operations need to mark the transaction version number (seq) on the corresponding row (i.e., the row in the data table).

[0196] Step 1404: In the database, the deletion version number is stored in the active memory table in memory. When the active memory table is full, it is converted into an inactive memory table and waits for the disk flushing operation to be performed.

[0197] Specifically, this solution addresses the issue of excessive memory consumption caused by storing version numbers entirely in memory by persisting the data to disk. Furthermore, considering the high cost of persisting data to disk, an LSM-tree mask architecture is provided to resolve this problem. The specific implementation method is as follows:

[0198] 1. LSM-tree mask is an LSM-tree architecture that includes an active in-memory table (mem), of which there is one and only one in a single database system, to provide high-concurrency read and write operations;

[0199] 2. Once the memory table is full, it is converted into a read-only inactive memory table (imm) and awaits flushing to disk.

[0200] Multiple imm instances may accumulate, and these accumulated imm instances are linked together in the order of their transformations to form a linked list.

[0201] Step 1406: Database, perform a flush operation on inactive memory tables.

[0202] After the imm file is written to disk, multiple files (FileStore) will be created. These files are merged in descending order of size to form multiple layers (i.e., L0, L1, L2 in Figure 9). Each file is divided into three information segments, as follows:

[0203] meta: Records the size of the file and the location of each information segment;

[0204] Data pages: Records all deleted records.<rowid,seq> Each deletion record is stored in the form of <row number, deletion version number>.

[0205] Level-trie: A type of location index, which can also be understood as a hash table, can locate a specific data page based on the row number, improving the efficiency of file seek (query) operations.

[0206] Based on this, writing an imm to disk can be seen as merging an imm with an empty FileStore. The specific disk flushing operation includes the following steps:

[0207] 1. Select the K files to be merged from the disk.

[0208] First, all files are scored to obtain a score for each file.

[0209] The principle for selecting filestores is to merge smaller filestores into larger ones, minimizing file fragmentation on disk. Since the key-value pair length in the LSM-tree mask is fixed, the number of key-value pairs in each filestore can represent the file size; and files are selected based on the following formula:

[0210] Where k represents the k files selected from n filestores, and storek This indicates the size of the i-th selected file; the file selection process must ensure that at least two files exist.

[0211] Secondly, there are two methods for selecting k files from n files:

[0212] (1) Starting from file i, select k files in sequence, that is, the set of files is {i,i+1,i+2,...,i+k}.

[0213] For example, if you have 5 files and want to select 3, you can choose the better combination from the following three options: 1, 2, 3; 2, 3, 4; 3, 4, 5.

[0214] (2) Perform an exhaustive search based on the permutation count to obtain multiple file components obtained through exhaustive search. Then choose the better one from them.

[0215] It should be noted that during the Flush operation, k filestores need to be selected. If k = 0, a new file is generated directly based on the inactive memory table; if k is not equal to 0, multiple existing files need to be selected to participate in the merge. The value of k needs to be calculated based on the merging algorithm to find the optimal solution.

[0216] 2. Merge the K files with the imm file to obtain a new file, thus completing the disk write operation.

[0217] Step 1408: Database. For the multiple files obtained after being written to disk, perform a file merging operation.

[0218] There are two ways to write data from an LSM-tree mask to disk:

[0219] (1) After mem is full, it is converted to imm. Then, the disk flushing operation will be called to flush it to the disk in the background and turn it into FileStore, which is the disk write operation in step 1406 above.

[0220] (2) When the number of FileSotre files on the disk reaches a certain amount, k files are selected from the n files and merged into one file according to a better strategy.

[0221] In fact, the first method can be seen as merging an imm file with an empty FileStore; therefore, the two methods of writing data to disk are collectively referred to as merge operations. The specific execution steps for the second merge operation are the same as those for the file merge operation in step 1406 above, and will not be elaborated upon here.

[0222] Step 1410: When the database receives a data read request for a data table, it determines the corresponding read version number for that data read request.

[0223] Step 1412: Database, based on the row number (e.g., 30) carried in the data read request, load the deletion version number from the disk using lsm-tree mask to determine whether the row data is visible.

[0224] Specifically, the steps to load the deletion version number from the disk are as follows:

[0225] 1. lsm-tree mask: Determine the min-max index from the level-trie index contained in each file, and filter all files based on the line number (30) to obtain the files that need to be iterated.

[0226] Specifically, to reduce the number of files that need to be iterated, the metadata of each file records the min-max index within that file, i.e., the minimum and maximum rowid. Therefore, during seek, files that do not need to be iterated are filtered out based on the row number (30). For example, if the rowid range of a file is [100, 500], seek(800) does not need to iterate over that file; however, due to the low-bound semantics of seek, seek(30) does need to query that file.

[0227] 2. For the files that need to be iterated, use the level-trie corresponding to each file to locate the page id corresponding to the key (i.e., line number 30).

[0228] 3. Based on the page ID, determine the corresponding data page from each file, and use binary search to further determine lower_bound(key).

[0229] Here, lower_bound(key) refers to the first position greater than or equal to key. For example, file0 corresponds to the position where rowid=32, while file1 corresponds to the position where rowid=30, and so on.

[0230] 4. Read the deletion version number starting from the lower_bound(key) position.

[0231] It should be noted that during the above iterative query, the n files are sorted using multi-way merge sort.

[0232] Step 1414: In the database, perform multi-level data merging on the deletion version number in memory, determine the minimum value of the deletion version number, and determine data visibility based on the minimum value and the read version number.

[0233] Merging multi-level data can be a merge sort process, where k levels of data require a k-way external sort. This k-way external sort can be understood as an external sorting operation on the k levels of data. The k levels of data can be understood as K ordered files, sorted in ascending order of rowid.

[0234] For multi-level data merging, the lsm-tree mask implementation uses a cached loser-tree to achieve k-way merging. The specific structure is as follows:

[0235] The disk contains k ordered files; while the DRAM contains two layers of cache and a loser-tree for sorting.

[0236] Among them, the L1 cache in the two-layer cache is a batch cache, which is used to load a batch of data from the disk at one time, such as 64KB, to reduce random I / O operations on the disk;

[0237] The L0 cache in the two-level cache is an array that records...<rowid,seq> Data pairs are used to load one key-value pair from the L1 cache at a time, primarily for deleting version numbers.<rowid,seq> Input into the loser-tree and fill in the gaps in the loser-tree's output;

[0238] The loser-tree is essentially an array of indices, with the top layer storing the index of the minimum value in the L0 cache. The loser-tree is used to efficiently sort this array. The top node of the loser-tree records the index of the minimum value in the array, where the minimum value refers to the rowid. For example, if the top node of the loser-tree records 4, then the minimum value is the fourth element of the array. After finding the minimum value, it is removed from the loser-tree, and then the next record is read from the file containing the fourth element of the array (i.e., the L0 cache). The loser-tree is then adjusted, and after adjustment, the top node of the loser-tree is the next minimum value.

[0239] After the LSM-tree mask outputs the minimum value of the loser-tree, the loser-tree will calculate the next minimum value in O(ln k) time. Since the loser-tree and L0 cache have a small amount of data (maximum K) and are stored contiguously together, they have the highest cache friendliness.

[0240] After outputting the minimum value, its index can be used to locate the position of the minimum value in the L0 cache. Then, a copy of the data is loaded from the corresponding L1 cache. If the L1 cache is already full, the next batch of data is loaded from the corresponding file into the L1 cache.

[0241] The usefulness of extracting the minimum value depends on the upper-level logic. For an LSM-tree mask, after extracting the minimum value, its sequence number (seq) can be calculated to determine whether the current row has been deleted.

[0242] It is important to emphasize that, in order to improve merging efficiency, lsm-tree mask supports multi-path parallel merging; that is, the multi-level data merging operation in step 1414 above is performed in a multi-path parallel manner, and the specific implementation steps are as follows.

[0243] 1. During the mask scan (delete version number) operation, the executor can create an lsm-tree iterator for each thread according to the configured concurrency, but different iterators can choose different scan start and end points.

[0244] Logically, the mask data is divided according to the number of threads, and different threads scan different parts. In the database executor implementation, each thread scans the mask in units of logical pack groups.

[0245] 2. Based on the group ID, the lsm-tree mask calculates the starting rowid and ending rowid of the group, thus locating the start and end points of the data to be read in each layer of the lsm-tree mask.

[0246] The group ID can be determined based on the data query requirements of the upper layer, which is the pack group ID of the data to be queried in the data table. The upper layer provides the row group ID, and the LSM-tree mask can calculate the start and end row IDs of this group ID. For example, if a group has 100 rows of data, then given a group ID = 9, we know that the start row ID of this group is 900 and the end row ID is 1000.

[0247] 3. Data location for LSM-tree masks uses a full-memory level-trie index, which can be completed in O(1) time without accessing the disk. Since different execution threads always scan different groups, mask scanning can be maximized for parallelism.

[0248] This Level-trie can be serialized and stored on disk, and then fully loaded into memory at startup. From a data structure perspective, the data structures of the files written to disk are static, thus eliminating concurrent read / write issues. Multi-threaded scan performance depends on disk bandwidth and the number of CPUs.

[0249] The active memtable in memory requires concurrent read and write operations. However, through the B+tree design with read-write separation mentioned above, data scanning and updating can be completely separated. As a result, the scanning performance in the lsm-tree mask has a linear speedup as the CPU increases, before reaching the disk bandwidth.

[0250] It should be noted that during the multi-level data merging process, the lsm-tree mask structure contains both active and inactive memory tables. Therefore, both active and inactive memory tables can be treated as a single file, and the memory tables and the files written to disk are treated equally during the merging process.

[0251] For example, if the upper-level logic needs to calculate the visibility of the group with group id=9 and group size=100, it initially creates a bitmap of size 100, all of which are 1, to indicate that it is fully visible.

[0252] Based on this group, we can calculate the starting rowid = 900, the ending rowid = 1000, and the scan mask has 5 records: {<903,4>,<920,8>,<932 10>,<950,5>}, where <903,4> represents rowid = 900 and seq = 4.

[0253] Assuming the read seq of this query transaction is 6, it can only see the two deleted records {<903,4>,<950,5>}, meaning it cannot see records with higher seq values. Then, the bitmap offset is calculated using the rowid: 903%100 = 3, 950%100 = 50. The 3rd and 50th bits of this bitmap are set to 0. This completes the bitmap generation process.

[0254] In addition, the lsm-tree mask in this solution can also perform incremental checkpoint operations during the efficient data marking and deletion process, as detailed below.

[0255] Checkpointing refers to taking a snapshot of the running data of the LSM-tree mask and saving it to disk. At startup, the data can be loaded from disk into memory to continue operation. The checkpointing mechanism needs to have the following characteristics: 1. Non-blocking: Checkpointing should not affect ongoing operations. This requires checkpoints to support non-blocking; 2. Transactional: Checkpoints should ensure the ACID properties of transactions.

[0256] Regarding the first point above, the lsm-tree mask needs to support an asynchronous checkpointing mechanism. Regarding the second point above, checkpoints need to not contain uncommitted data or invisible data. Here, "committed" refers to the basic concept of ACID transactions in databases; if a transaction is not committed, the data within it cannot be recorded in the checkpoint.

[0257] However, the following two problems arise when implementing the checkpoint mechanism, given the above two characteristics:

[0258] First, checkpointing is performed asynchronously in the background, so foreground operations still write to the lsm-tree mask during checkpointing.

[0259] Secondly, according to the constraints of the LSM-tree architecture paradigm, data that has been written to disk cannot be changed; if the data written to disk contains future data, then checkpointing operations cannot be performed.

[0260] To address the above issues, this solution introduces a dual LSM-tree design with a copy-on-write mechanism in the LSM-tree mask. The LSM-tree mask is split into a checkpoint tree and a delta tree at runtime.

[0261] The checkpoint tree is generated in the background, strictly guaranteeing the ACID semantics of IMCI transactions. It is controlled by the checkpoint sequence and will not contain future data.

[0262] The delta tree is used to record the incremental part of the LSM-tree mask, which contains data greater than the latest checkpoint. It is mainly used to ensure that the data in the memtable can be flushed to the disk in a timely manner to reduce the memory usage of the LSM-tree mask. The delta tree is exclusive to each AP RO instance and its data is automatically cleaned up upon restart.

[0263] The specific steps for performing this incremental checkpoint operation are as follows:

[0264] 1. When lsm-tree mask is started, the checkpoint tree and delta tree share the same data.

[0265] 2. As the front end writes data, the two trees will gradually split, forming two independent checkpoint trees and delta trees.

[0266] The checkpoint tree generates data according to strict ACID transaction semantics; the delta tree flushes the memtable to disk as quickly as possible to free up memory, in accordance with the principle of saving memory.

[0267] Specifically, after splitting into two lsm-trees, the two lsm-trees will share some file data; and the operations described in steps 1402 to 1414 above, such as "data persistence and merging, parallel scanning, and multi-level data merging" performed by the lsm-tree mask, will be performed in each lsm-tree.

[0268] 3. Each time a checkpoint is performed in the database, the lsm-tree mask traverses the delta tree based on the input checkpoint sequence number, filters out outdated data and data that is no longer needed, and stores the remaining data in the checkpoint tree.

[0269] Among them, the leading data is determined by seq. If the current checkpoint seq = 100, then all data with seq greater than 100 are leading data.

[0270] Data that is no longer needed is determined by the upper-level logic. For example, if data with rowid in [100, 500] is no longer needed in a practical application, then this data is no longer needed, and lsm-mask can delete it directly.

[0271] It should be noted that for data that already exists in the checkpoint-tree but is no longer needed, the checkpoint-tree will perform a merge operation as needed to delete it from the checkpoint-tree.

[0272] 4. When reading data from the database, always read the data stored in the delta tree.

[0273] It's important to note that the checkpoint tree is primarily used by the storage engine to load data from a consistent transaction point at startup. This transaction point is determined by the checkpoint, which serves to identify a safe point for persisting the data to disk. The data loaded during the parallel scan process described above also includes all data prior to the checkpoint.

[0274] At this point, the LSM-tree mask firstly ensures that reading will not see future data and will not miss any data; secondly, the checkpoint always records incremental data, which significantly reduces the I / O consumption of writing data to disk compared to the full disk flushing scheme of the full memory array.

[0275] As can be seen from the above embodiments, the data processing methods in one or more embodiments of this disclosure provide an efficient data marking and deletion scheme for HTAP databases. This scheme proposes an LSM-tree mask framework structure, which can store masks for different rows. Different transactions write mask data to different rows. Furthermore, this scheme can ensure that these transactions write independently and that the writes do not block each other.

[0276] This solves the two problems of excessive mask memory usage and high data write-to-disk cost without sacrificing the performance of the full memory mask. It can significantly reduce the memory usage of the database instance without sacrificing performance, greatly reduce the risk of system crash due to insufficient memory, and improve the overall stability of the database system.

[0277] Furthermore, it is important to emphasize that the lsm-tree mask, through the execution steps 1402 to 1414 described above, avoids the shortcomings of other solutions, as detailed below:

[0278] 1. The row-level versioning scheme ensures that concurrent transactions updating masks do not block each other, resulting in high concurrent update performance. It avoids concurrency contention issues at the table and block levels.

[0279] 2. During reading, only row numbers are merged, and the row numbers are of fixed length and can be parallelized through row groups, so the merging efficiency is very high, avoiding the high-cost merging problem of using pk, table-level and block-level bitmap marking in the industry.

[0280] 3. Only the memory table needs to be stored in memory, and its size is configurable. Most of the data is written to disk, avoiding the high memory usage problem of the full memory array version number scheme.

[0281] 4. Mask checkpoint operations only record incremental updates, avoiding the high I / O consumption of full disk flushing used in the full memory array version number scheme.

[0282] Based on the aforementioned technical effects, this disclosure provides an efficient data marking and deletion scheme for HTAP databases. It can resolve the contradiction between efficient updates, fast queries, and resource consumption in integrated online transaction analysis database products, achieving extremely low memory usage while meeting the system's requirements for efficient updates and fast queries. The advantages of this scheme are:

[0283] (1) Row-level transaction-granularity mark-delete strategy to ensure that concurrent transactions in the database do not block each other, that is, to ensure that concurrent updates of the mask do not block each other in the case of concurrent transactions.

[0284] (2) Parallel scanning mechanism under the lsm-tree architecture to achieve multi-core linear speedup of labeled data.

[0285] (3) Dual LSM-tree strategy to achieve transaction-level checkpoint without blocking speed.

[0286] (4) Lightweight row number merging and row group-level parallelization enable highly efficient query operations and have a linear speedup under multi-core conditions.

[0287] (5) Dual LSM-tree checkpoint strategy to achieve lightweight and non-blocking transaction-level checkpoints.

[0288] Corresponding to the above method embodiments, this disclosure also provides a data processing apparatus embodiment, which is applied to the storage engine of a database. The storage engine uses row numbers to record the write version number and delete version number of each row. The storage engine includes active memory tables and inactive memory tables, including:

[0289] The data writing module is configured to write deletion marker data to the active memory table, wherein the deletion marker data includes the row number of the deleted data and the deletion version number, so that in response to a data query operation, the data visibility of the corresponding row number can be determined based on the deletion marker data;

[0290] The memory table conversion module is configured to convert the active memory table into an inactive memory table if the remaining space of the active memory table is less than or equal to a preset space threshold.

[0291] The memory table write module is configured to write deletion marker data from the inactive memory table to a persistent storage device in response to a disk write operation.

[0292] Optionally, the memory table writing module is further configured to:

[0293] A first preset number of files are selected from the files in the persistent storage device, and the first preset number of files are merged with the deletion mark data in the inactive memory table to generate a new file.

[0294] Optionally, the memory table writing module is further configured to:

[0295] Based on file size, a first preset number of files are selected from the files in the persistent storage device.

[0296] Optionally, the data processing device further includes a file generation module, configured to:

[0297] A second preset number of files are selected from the files in the persistent storage device, and the second preset number of files are merged to generate a new file, wherein the file is the file in the persistent storage device that stores the deletion mark data.

[0298] Optionally, the data processing device further includes a file data deletion module, configured to:

[0299] If the deletion marker data is visible in any file in the persistent storage device, then the deletion version number in that file is removed.

[0300] Optionally, the data processing device further includes a data replication module, configured to:

[0301] In response to a checkpoint operation, incremental data in the data merge tree is copied to the checkpoint data tree, wherein the incremental data is deletion marker data between the last checkpoint operation and the currently executed checkpoint operation.

[0302] Optionally, the data replication module is further configured to:

[0303] Filter out the target data from the incremental data to obtain the data to be copied. The target data includes the deletion marker data generated after this checkpoint operation and / or the deletion marker data that meets the data deletion conditions.

[0304] The data to be copied is copied to the checkpoint data tree.

[0305] Optionally, the data processing device further includes a data loading module, configured to:

[0306] In response to a data query operation, determine the start row number and end row number corresponding to the data query operation;

[0307] The target object is determined from the data merging tree based on the start line number and the end line number, wherein the target object includes one or more of the active memory table, the inactive memory table, and the file;

[0308] Determine the initial data loading row number from the target object based on the starting row number;

[0309] The deletion marker data is loaded starting from the initial data loading row number, and the data visibility of the corresponding row number is determined based on the deletion marker data.

[0310] Optionally, the data loading module is further configured to:

[0311] Determine the row group corresponding to the data query operation, and set the starting row number and ending row number of the row group as the starting row number and ending row number of the data query operation. The row group is determined based on multiple row numbers.

[0312] Optionally, the data loading module is further configured to:

[0313] A preset number of deletion marker data are loaded into memory from the initial data loading row number;

[0314] The deletion marker data in the memory is sorted to obtain ordered deletion marker data, and the data visibility of the corresponding row number is determined based on the ordered deletion marker data.

[0315] When the data loading conditions are met, the end line number of the previous data loading is used as the new initial data loading line number, triggering the step of loading a preset number of deletion marker data from the initial data loading line number into memory.

[0316] Optionally, the data loading module is further configured to:

[0317] The deletion marker data of a first preset number is loaded from the initial data loading line number and stored in the first cache;

[0318] Load a second preset amount of deletion marker data into the second cache from the first cache;

[0319] Accordingly, sorting the deletion marker data in the memory to obtain ordered deletion marker data includes:

[0320] The deletion marker data is loaded from the second cache into the loser tree, and the loser tree outputs the ordered deletion marker data.

[0321] In the data processing apparatus provided in one or more embodiments of this disclosure, during the processing of deletion marker data, newly generated deletion marker data is written into the active memory table in the data merging tree. When it is determined that the active memory table is full, the active memory table is converted into an inactive memory table. Furthermore, in response to the disk write operation, the inactive memory table is written into a persistent storage device to generate a new file. By writing the deletion marker data in memory into the persistent storage device, the problem of deletion marker data occupying a large amount of memory resources is avoided, and memory resources can be used normally.

[0322] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0323] Figure 15 shows a structural block diagram of a computing device 1500 according to an embodiment of the present disclosure. The components of the computing device 1500 include, but are not limited to, a memory 1510 and a processor 1520. The processor 1520 is connected to the memory 1510 via a bus 1530, and a database 1550 is used to store data.

[0324] The computing device 1500 also includes an access device 1540, which enables the computing device 1500 to communicate via one or more networks 1560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1540 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0325] In one embodiment of this disclosure, the aforementioned components of the computing device 1500, as well as other components not shown in FIG. 15, may also be connected to each other, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 15 is merely for illustrative purposes and is not intended to limit the scope of this disclosure. Those skilled in the art can add or replace other components as needed.

[0326] The computing device 1500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1500 can also be a mobile or stationary server.

[0327] The processor 1520 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0328] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the data processing method embodiments.

[0329] An embodiment of this disclosure also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0330] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the data processing method embodiments.

[0331] An embodiment of this disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0332] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0333] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0334] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0335] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this disclosure.

[0336] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0337] The preferred embodiments disclosed above are merely illustrative of this disclosure. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments of this disclosure. These embodiments are selected and specifically described in this disclosure to better explain the principles and practical applications of the embodiments of this disclosure, thereby enabling those skilled in the art to better understand and utilize this disclosure. This disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method applied to a database storage engine, wherein the storage engine uses row numbers to record the write version number and delete version number of each row, and the storage engine includes an active memory table and an inactive memory table, comprising: The deletion marker data is written to the active memory table, wherein the deletion marker data includes the row number of the deleted data and the deletion version number, so that in response to a data query operation, the data visibility of the corresponding row number can be determined based on the deletion marker data; If the remaining space of the active memory table is less than or equal to a preset space threshold, then the active memory table is converted into an inactive memory table. In response to the disk write operation, the deletion marker data in the inactive memory table is written to the persistent storage device.

2. The method according to claim 1, wherein writing the deletion marker data in the inactive memory table to the persistent storage device comprises: A first preset number of files are selected from the files in the persistent storage device, and the first preset number of files are merged with the deletion mark data in the inactive memory table to generate a new file.

3. The method according to claim 2, wherein selecting a first preset number of files from the files of the persistent storage device comprises: Based on file size, a first preset number of files are selected from the files in the persistent storage device.

4. The method according to any one of claims 1 to 3, further comprising: A second preset number of files are selected from the files in the persistent storage device, and the second preset number of files are merged to generate a new file, wherein the file is the file in the persistent storage device that stores the deletion mark data.

5. The method according to any one of claims 2 to 4, further comprising: If the deletion marker data is visible in any file in the persistent storage device, then the deletion version number in that file is removed.

6. The method according to any one of claims 1 to 5, further comprising: In response to a checkpoint operation, incremental data in the data merge tree is copied to the checkpoint data tree, wherein the incremental data is deletion marker data between the last checkpoint operation and the currently executed checkpoint operation.

7. The method according to claim 6, wherein copying the incremental data in the data merging tree to the checkpoint data tree comprises: Filter out the target data from the incremental data to obtain the data to be copied. The target data includes the deletion marker data generated after this checkpoint operation and / or the deletion marker data that meets the data deletion conditions. The data to be copied is copied to the checkpoint data tree.

8. The method according to any one of claims 1 to 7, further comprising: In response to a data query operation, determine the start row number and end row number corresponding to the data query operation; The target object is determined from the data merging tree based on the start line number and the end line number, wherein the target object includes the active memory table, the inactive memory table, and one or more of the files; Determine the initial data loading row number from the target object based on the starting row number; The deletion marker data is loaded starting from the initial data loading row number, and the data visibility of the corresponding row number is determined based on the deletion marker data.

9. The method according to claim 8, wherein determining the start row number and end row number corresponding to the data query operation includes: Determine the row group corresponding to the data query operation, and set the starting row number and ending row number of the row group as the starting row number and ending row number of the data query operation. The row group is determined based on multiple row numbers.

10. The method according to claim 8 or 9, wherein loading the deletion marker data starting from the initial data loading row number and determining the data visibility of the corresponding row number based on the deletion marker data comprises: A preset number of deletion marker data are loaded into memory from the initial data loading row number; The deletion marker data in the memory is sorted to obtain ordered deletion marker data, and the data visibility of the corresponding row number is determined based on the ordered deletion marker data. When the data loading conditions are met, the end line number of the previous data loading is used as the new initial data loading line number, triggering the step of loading a preset number of deletion marker data from the initial data loading line number into memory.

11. The method according to claim 10, wherein the memory includes a first cache and a second cache, and loading a preset number of deletion marker data from the initial data loading line number into the memory includes: The deletion marker data of a first preset number is loaded from the initial data loading line number and stored in the first cache; Load a second preset amount of deletion marker data into the second cache from the first cache; Accordingly, sorting the deletion marker data in the memory to obtain ordered deletion marker data includes: The deletion marker data is loaded from the second cache into the loser tree, and the loser tree outputs the ordered deletion marker data.

12. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.

14. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.

Images