A method and apparatus for incremental merge in a database system

Abstract

This specification provides an embodiment of a method for incremental merging in a database system. The database system has a storage architecture of Log Structure Merge Tree (LSM-Tree) and includes data nodes and several compute nodes. The method is executed by a first compute node among the compute nodes and includes: retrieving first incremental data and a first commit log from an active memory table (MEMTable); the first commit log records the generation process of the first incremental data. The first incremental data is persisted to a local first data file by performing a first merging operation (compaction). The first commit log is uploaded to the data node, causing the data node to update the memory checkpoint of the first compute node.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and more particularly to a method and apparatus for incremental merging in a database system. Background Technology

[0002] A distributed database is a database that runs across multiple computers. It distributes data across multiple computer nodes, which can be in a cluster environment in the same geographical location or distributed across multiple data centers in different regions. Compared to traditional single-node databases, distributed databases effectively overcome the storage capacity limitations and computing performance bottlenecks of single nodes, significantly improving the scalability, availability, and access performance of the database system. They also support large-scale data storage and processing, and are therefore widely used in fields such as e-commerce, finance, and the Internet of Things that require massive data storage and high-performance data access.

[0003] In terms of data storage architecture, Log Structure Merged Tree (LSM-Tree), as a highly efficient write-optimized storage model, has been adopted by numerous distributed database systems. LSM-Tree significantly improves data write throughput by converting random writes into sequential append writes. Its core mechanism is to temporarily store incremental data in memory; when the incremental data occupies a certain amount of memory, memory is frozen and a merge operation is performed to persist the in-memory data into multi-level data files on disk. However, with the continuous expansion of data scale and the increasing computational load, the traditional LSM-Tree architecture has gradually revealed the following shortcomings in distributed database systems using shared storage models.

[0004] Regarding incremental data merging, traditional LSM-Trees typically freeze and persist incremental data to data files only when the incremental data occupies a certain amount of memory. The triggering frequency is limited by memory capacity and I / O bandwidth. When a new node is added or a node is recovered in the database system, all incremental data accumulated since the last persistence needs to be replayed. If the amount of data in memory that has not yet been persisted to disk is too large, it will significantly prolong the data replay process and affect the service continuity of the database system.

[0005] In terms of full data merging, traditional LSM-Trees typically delegate this computationally intensive task to compute nodes, which can easily compete with foreground transactions for CPU, memory, and I / O resources, leading to increased query latency and decreased throughput in the database system. In shared storage mode, multiple compute nodes may concurrently perform full merges; the lack of coordination mechanisms can easily cause write conflicts or data inconsistencies, hindering system performance and stability.

[0006] Although shared storage architecture improves node resource utilization by decoupling storage and computing resources, the aforementioned issues limit the further application of distributed database systems in cloud-native environments.

[0007] Therefore, we hope to provide a solution that optimizes the incremental and full merge execution processes for distributed database systems with shared storage architectures through technical means. Without affecting the performance of front-end transactions, this solution should, on the one hand, enable rapid incremental merge execution, and combined with the data sharing mechanism of the shared storage architecture, reduce the amount of data replayable during incremental data transfer, thereby improving the service continuity and elastic scalability of the database system; on the other hand, it should reduce competition for computing resources during the full merge process, ensuring the performance and stability of the database system. Summary of the Invention

[0008] This specification describes one or more embodiments of a method and apparatus for incremental merging and full merging in a database system, which can solve the above-mentioned technical problems.

[0009] According to a first aspect, a method for incremental merging in a database system is provided, wherein the database system has a storage architecture of Log Structure Merge Tree (LSM-Tree) and includes data nodes and a plurality of compute nodes. The method is executed by a first compute node among the plurality of compute nodes and includes:

[0010] The first incremental data and the first commit log are obtained from the active memory table MEMTable; the first commit log is used to record the generation process of the first incremental data.

[0011] By performing the first compaction operation, the first incremental data is persisted to the local first data file.

[0012] The first commit log is uploaded to the data node, causing the data node to update the memory checkpoint of the first compute node.

[0013] According to the second aspect, a method for incremental merging in a database system is provided. The database system has a storage architecture of Log Structure Merge Tree (LSM-Tree) and includes data nodes and several compute nodes. The method is executed by a first compute node among the compute nodes, which performs a reconstruction operation on a first replica. The method includes:

[0014] Obtain the memory checkpoint corresponding to the first replica from the data node. The memory checkpoint is generated by the method described in the first aspect and is used to indicate the data starting point for replaying incremental data of the first replica into the memory table MEMTable.

[0015] Obtain the first commit log from the data node; the first commit log is used to record the generation process of the first incremental data, which is the incremental data generated by the first replica after the memory checkpoint.

[0016] Replay the first commit log to store the first incremental data into the memory table.

[0017] According to a third aspect, a method for performing a full merge in a database system is provided. The database system has a storage architecture of a log structure merge tree (LSM-Tree) and includes several data nodes and several compute nodes. The method is executed by a first data node among the several data nodes and includes:

[0018] The first merge task is obtained from the metadata service shared by all nodes. The first merge task is generated by the first compute node and is used to instruct a full merge compaction to be performed on several partitions. The first compute node is the primary replica management node of the several partitions.

[0019] Based on the data files corresponding to the aforementioned partitions, a full merge is performed to obtain the target data file.

[0020] The execution status of the first merge task in the metadata service is updated so that the computing nodes associated with the plurality of partitions update the relevant cache data according to the target data file.

[0021] According to the fourth aspect, a method for performing a full merge in a database system is provided. The database system has a storage architecture of a log structure merge tree (LSM-Tree) and includes data nodes and several compute nodes. The method includes:

[0022] The first computing node among the plurality of computing nodes responds to the full merge task initiated by the root service of the database system and generates a first merge task for the plurality of first partitions; the first computing node is the primary replica management node of the plurality of first partitions.

[0023] The first computing node writes the first merging task into the metadata service shared by all nodes.

[0024] The data node obtains the first merging task through the metadata service, and performs a full merging compaction on the first merging task to obtain the target data file.

[0025] The data node updates the execution status of the first merging task in the metadata service.

[0026] In response to the update of the execution status, the second computing node associated with the plurality of first partitions updates the relevant cache data according to the target data file.

[0027] According to the fifth aspect, this specification provides an apparatus for incremental merging in a database system, wherein the database system has a storage architecture of a log structure merge tree (LSM-Tree) and includes data nodes and a plurality of compute nodes; the apparatus is deployed on a first compute node among the plurality of compute nodes and includes:

[0028] The acquisition module is configured to acquire first incremental data and first commit log from the active memory table MEMTable; the first commit log is used to record the generation process of the first incremental data.

[0029] The merging module is configured to persist the first incremental data to a local first data file by executing a first merging operation, compaction.

[0030] The upload module is configured to upload the first commit log to the data node, so that the data node updates the memory checkpoint of the first computing node.

[0031] According to a sixth aspect, this specification provides an apparatus for incremental merging in a database system, the database system having a storage architecture of Log Structure Merge Tree (LSM-Tree), the database system including data nodes and a plurality of compute nodes; the apparatus is deployed on a first compute node among the plurality of compute nodes, the first compute node being used to perform a reconstruction operation of a first replica. The apparatus includes:

[0032] The acquisition module is configured to acquire the memory checkpoint corresponding to the first replica from the data node. The memory checkpoint is generated using the device described in the fifth aspect and is used to indicate the data starting point for replaying incremental data of the first replica in the memory table MEMTable.

[0033] The acquisition module is further configured to acquire a first commit log from the data node; the first commit log is used to record the generation process of the first incremental data, and the first incremental data is the incremental data generated by the first replica after the memory checkpoint.

[0034] The replay module is configured to replay the first commit log in order to store the first incremental data into the memory table.

[0035] According to the seventh aspect, this specification provides an apparatus for performing a full merge in a database system, wherein the database system has a storage architecture of a log structure merge tree (LSM-Tree) and includes several data nodes and several computing nodes. The apparatus is deployed on a first data node among the several data nodes and includes:

[0036] The acquisition module is configured to acquire a first merge task from the metadata service shared by each node. The first merge task is generated by a first compute node and is used to instruct a full merge compaction to be performed on several partitions. The first compute node is the master replica management node of the several partitions.

[0037] The merging module is configured to perform a full merge based on the data files corresponding to the partitions to obtain the target data file.

[0038] The update module is configured to update the execution status of the first merge task in the metadata service, so that the computing nodes associated with the plurality of partitions update the relevant cache data according to the target data file.

[0039] According to aspect eight, this specification provides a system for performing full data merging in a database system, wherein the database system has a storage architecture of a log structure merge tree (LSM-Tree) and includes data nodes and several compute nodes. The system includes:

[0040] The first compute node is configured to generate a first merge task for several first partitions in response to a full merge task initiated by the root service of the database system; the first compute node is the primary replica management node of the several first partitions.

[0041] The first computing node is further configured to write the first merging task into a metadata service shared by all nodes.

[0042] The data node is configured to obtain the first merging task through the metadata service, and perform a full merging compaction on the first merging task to obtain the target data file.

[0043] The data node is also configured to update the execution status of the first merging task in the metadata service.

[0044] The second computing node is configured to update relevant cached data based on the target data file in response to an update of the execution status; the second computing node is associated with the plurality of first partitions.

[0045] According to a ninth aspect, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the methods described in the first, second, third, or fourth aspects.

[0046] According to a tenth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect, the second aspect, the third aspect, or the fourth aspect.

[0047] In summary, the embodiments provided in this specification present a method for incremental and full merging in a database system. In a distributed database system employing a shared storage architecture, this method can accelerate the generation of memory checkpoints and increase the frequency of incremental data writes to disk in the MEMTable, thereby enhancing the continuous service capability of the database system during fault recovery or elastic scaling and reducing overhead. Furthermore, by leveraging the characteristics of the shared storage architecture, the resource-intensive full merging operation can be moved from the computing nodes responsible for handling online transactions to the data nodes, completely separating computing and storage resources. This avoids the problem of computing nodes competing for computing resources with foreground transactions due to the execution of full merging tasks. Through metadata services for task coordination and state synchronization, decoupling and asynchronous collaboration between computing nodes and data nodes are achieved, significantly improving the elasticity and sustainable service capability of the distributed database system under the shared storage architecture. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0049] Figure 1 This is a schematic diagram of a typical LSM-Tree storage architecture disclosed in this specification;

[0050] Figure 2 This is a schematic diagram of an exemplary distributed database implementation architecture disclosed in this specification;

[0051] Figure 3 This specification provides an exemplary implementation architecture for a distributed database employing a shared storage model.

[0052] Figure 4A This specification discloses an implementation framework for incremental merging in a database system.

[0053] Figure 4BThis specification discloses an implementation framework for performing a full merge in a database system.

[0054] Figure 5 This is a flowchart illustrating a method for incremental merging in a database system according to an embodiment of this specification.

[0055] Figure 6 This is a flowchart illustrating a method for performing a full data merge in a database system according to an embodiment of this specification.

[0056] Figure 7 This is a schematic diagram of an apparatus for incremental merging in a database system according to an embodiment of this specification;

[0057] Figure 8 This is a schematic diagram of an apparatus for incremental merging in a database system according to an embodiment of this specification;

[0058] Figure 9 This is a schematic diagram of an apparatus for performing a full data merge in a database system according to an embodiment of this specification;

[0059] Figure 10 This is a schematic diagram of a system for performing a full merge in a database system according to an embodiment of this specification. Detailed Implementation

[0060] The solutions provided in the embodiments of this specification will now be described with reference to the accompanying drawings.

[0061] Currently, in many enterprises, data analysis and operational activities are all based on databases as their data foundation. Databases can efficiently organize, store, and manage massive amounts of data. Through flexible query interfaces and transaction mechanisms, they provide efficient data read and write services for upper-layer applications while ensuring data consistency and reliability.

[0062] From a database architecture perspective, databases can employ an append-only write strategy to handle new data. This means that insert, update, and delete operations performed on a database instance are all stored as newly added data, and once stored, the data is not modified. For example, data insertion can be achieved by directly appending new data records, updating historical data can be achieved by appending new version data records, and deleting historical data can be achieved by appending data to delete marker records.

[0063] At the data storage level, besides organizing data in data pages and persisting it to storage media, databases can also use log-structured storage structures for data persistence. Typically, an LSM-Tree (Log Structured Merge Tree) storage architecture can be used. Under this architecture, new data written to the database is first sequentially appended to a MEMTable in memory, forming incremental data. This incremental data is then persisted to storage media in an immutable form (Immutable MEMTable), creating multiple SSTable (Sorted String Table) files generated based on time order. In other words, the MEMTable resides in memory to store incremental data, providing read and write operations. When the space occupied by the incremental data stored in the MEMTable reaches a certain threshold, the incremental data in the MEMTable is frozen, forming an Immutable MEMTable, and further persisted to SSTable files. SSTables can be stored on non-volatile storage media for storing static data. Unlike RAM, non-volatile storage media features data retention after power failure, large storage capacity at relatively low cost, and high sequential write bandwidth but high random read / write latency. In this storage architecture, all data insertion, update, and deletion operations can be transformed into new data records and ultimately appended to the SSTable file, aligning with the append-only write strategy described above. In some practices, the SSTable file can be further divided into several fixed-size blocks at the logical level, with each block containing several data records.

[0064] The basic data unit for database operations can be the SSTable file described above, or the blocks that make up the SSTable file. In some embodiments of this specification, data blocks will be used to represent the basic data unit for database operations, and their specific meaning may vary depending on the database architecture design in the actual application.

[0065] For a concrete implementation example of the LSM-Tree database described above, please refer to [link / reference]. Figure 1 The diagram shows a typical LSM-Tree storage architecture. Database systems using this architecture can transform random write operations into batch sequential write operations, greatly improving data writing speed.

[0066] See Figure 1Data updates over a period of time (corresponding to write operations shown in the attached diagram) are persisted to a log file (corresponding to the Write-Ahead Log (WAL) shown in the attached diagram) in a sequential manner and written to a data structure in memory (corresponding to the MEMTable shown in the attached diagram). When the amount of data in the MEMTable exceeds a certain threshold, the MEMTable is frozen and transformed into an immutable MEMTable (corresponding to the ImmutableMEMTable shown in the attached diagram). Simultaneously, to avoid blocking database write operations, a new MEMTable is generated to respond to subsequent data writes. Next, background tasks in the database system persist the data in the Immutable MEMTable in memory (corresponding to the flush operation shown in the attached diagram) to the SSTable file on disk without blocking the processing of database foreground tasks. In this way, random write operations performed by the front-end system on different data in the database system can be batch-flushed to the SSTable on disk. This flushing process is a sequential append write to the SSTable, thus transforming the originally scattered random write operations into batch, continuous sequential write operations, significantly improving data write efficiency.

[0067] Please refer to the appendix for further details. Figure 1 The SSTables on disk are organized in a multi-level structure (the example in the attached diagram includes L0, L1, and L2, three levels). The number of levels can be set according to specific needs. Generally, the total capacity of the SSTables at each higher level is smaller than that at the next lower level. The L0 level SSTable is usually generated directly by flushing the Immutable MEMTable. Other levels (e.g., L1, L2) of SSTables are generated through compaction operations. Specifically, when the data capacity of the L0 SSTable reaches or approaches its limit, the data of the L0 SSTable can be written to the L1 SSTable through compaction. Similarly, when the data capacity of the L1 SSTable reaches its limit, the data of the L1 SSTable can be written to the L2 SSTable through compaction, and so on. In this way, the data stored in the SSTable at each higher level will be newer than that at the next lower level. Overall, the newest data is stored in memory, the second newest data is stored in the L0 SSTable, and as compaction operations gradually migrate to the next lower level, the oldest data is stored in the lowest level SSTable.

[0068] Please refer to the appendix for further details. Figure 1In LSM-Tree databases, the execution of read operations is also closely related to the aforementioned multi-level storage architecture. When processing a read operation, the database system first queries the MEMTable in memory. If no match is found, it continues to query the Immutable MEMTable in memory. If no matching data is found in memory, the system searches the multi-level SSTables on disk, typically starting from a certain level (e.g., L0 level) and traversing level by level down (e.g., from L0 to L1 level, then to L2 level) until matching data is found.

[0069] The above is a detailed introduction to the LSM-Tree storage architecture in databases. Database systems based on this storage architecture can convert random data writes into sequential append writes and reorganize data using background compression and merging operations, thereby achieving extremely high data write throughput. It is particularly suitable for carrying continuous writes of massive amounts of data and is widely used in data write-intensive scenarios such as time-series data recording and operation log archiving.

[0070] Next, we will first introduce the implementation architecture of distributed databases, then the typical implementation architecture of distributed databases under shared storage mode, and finally, based on the above content, explain the technical problems that distributed databases face in fault recovery or replica loading scenarios under shared storage mode. It should be noted that this specification... Figure 2 or Figure 3 The distributed database architecture shown is only an example, and the number of nodes (including compute nodes and data nodes), partitioning, and region affiliation are not limiting factors. In practice, these can be flexibly configured according to needs.

[0071] Figure 2 An exemplary distributed database implementation architecture is shown. See also... Figure 2 A distributed database system consists of several nodes, each typically a single physical machine. These nodes often belong to different regions, with each node belonging to one region. For example... Figure 2 The diagram shows region A, which contains nodes A1 and A2, and region B, which contains nodes B1 and B2. A region is a logical concept, typically representing a set of nodes with similar network conditions or geographical locations. That is, a region can have different meanings depending on the deployment model. For example, when a database system is deployed in a data center, nodes in a region can be machines belonging to the same rack or machines bridged on the same switch. When a database system is deployed in multiple data centers, each region can correspond to one data center, and the nodes in a region are the machines deployed within that data center.

[0072] In some distributed databases, data in a table can be split (horizontally partitioned) into multiple data shards according to certain partitioning rules. Each data shard is a table partition, or simply partition (or Tablet). Any row of data in a table belongs to one and only one partition. Multiple partitions of a data table can be distributed and stored across multiple different nodes. Figure 2 For example, data table 1 is divided into 4 partitions, data table 2 is divided into 2 partitions, and they are distributed and stored on various nodes.

[0073] In practical applications, to achieve high availability of a database system, a partition can have multiple replicas. Typically, these replicas are distributed across multiple different regions, and only one of these replicas can accept data modification operations; this primary replica is the main replica, and the others are secondary replicas. Figure 2 In the example shown, the rounded rectangle represents the primary copy of the partition, and the parallelogram represents the secondary copy of the partition.

[0074] The node hosting the primary replica can be considered the master node, and the nodes hosting the secondary replicas can be considered secondary nodes. Once the node hosting the primary replica begins providing services, user data update operations will generate corresponding logs / write-ahead logs on that node. These logs are then synchronized to all secondary replica nodes based on a distributed consensus protocol (e.g., Raft, Paxos), ensuring data consistency across replicas. When the node hosting the primary replica fails, the database system can automatically initiate an election through the consensus protocol to select a new master node from the surviving secondary nodes to continue providing services, achieving automatic fault recovery and continuous service availability.

[0075] Based on the above description of the distributed database implementation architecture, it is clear that distributed database systems achieve high availability and distributed processing through data sharding and multi-replica deployment. However, traditional distributed database systems often adopt a shared-nothing (SN) architecture, where each node needs to store a complete copy of the data. This not only leads to data redundancy and increased storage costs, but also necessitates complex data redistribution operations during node expansion or fault recovery, which is time-consuming and impacts the business continuity of the database system. Furthermore, during node fault recovery, complete data shards need to be synchronized from other nodes, resulting in low efficiency. To address this, the industry has proposed a shared storage architecture for distributed database systems. This architecture decouples data storage functions from nodes, forming a unified, shareable data storage layer (hereinafter referred to as the shared storage layer). This allows nodes to focus on computational tasks without the responsibility of data persistence and synchronization, thus achieving stateless characteristics. While effectively reducing data redundancy, it also improves elastic scalability.

[0076] Figure 3 An exemplary implementation architecture of a distributed database system employing a shared storage model is shown. See also... Figure 3 From an architectural perspective, the database system can be divided into a compute node layer and a shared storage layer. Additionally, it can include a metadata service layer and a log service layer. The log service layer and metadata service layer can serve as shared layers between multiple compute nodes and data nodes, achieving data sharing and synchronization based on a distributed consensus protocol. From an engineering implementation perspective, the log service layer and metadata service layer can be implemented as sub-modules within the shared storage layer, or they can be implemented as separate modules; this specification does not limit this approach. The aforementioned layers can be deployed independently and work collaboratively through standardized interfaces to achieve separation of computation and storage in the database system.

[0077] The compute node layer can consist of several stateless compute nodes, responsible for handling computationally intensive tasks such as querying and transaction execution. The shared storage layer provides persistent data storage and sharing services, ensuring data consistency. Specifically, the shared storage layer can be implemented using object storage services, which can be a distributed, scalable storage cluster storing database data based on an LSM-Tree write strategy. For any tenant, only complete copies of data and logs need to be stored in the shared storage layer to reduce data redundancy and lower storage costs. Simultaneously, to improve the performance of the database system in Transaction Processing (TP) scenarios, each compute node in the compute node layer can store hot data of partitions in its local cache, reducing dependence on access to the shared storage layer. Under this architecture, due to the stateless nature of the compute nodes, when scaling up or recovering from a fault, there is no need for data redistribution; only newly added data that has not yet been persisted in the shared storage layer needs to be retrieved from other nodes. This significantly improves the efficiency of node addition or fault recovery, reducing the impact on the business continuity of the database system.

[0078] Continue reading Figure 3 In a distributed database system employing a shared storage model, compute nodes can be divided into read-write compute nodes (RW nodes) and read-only compute nodes (RO nodes) based on their read and write permissions to replicas. These two types of nodes work together to provide high-concurrency transaction processing and query services for partitioned data within the replicas. Figure 3 In the diagram, rounded rectangles represent compute nodes managing primary replicas, and parallelograms represent compute nodes managing secondary replicas; a replica can include several partitions of data. Figure 3 In this context, the letter P represents partition data, such as P1, P2, etc. Compute nodes manage replicas using Log Streams (LS). It's understandable that since a compute node can manage multiple partitions, it may be classified as a Replica-W or Replica-O node, depending on whether it manages the replica type of a particular partition. In other words, the RW / RO attribute is determined by the partition replica type. For example, if a compute node manages the primary replica of partition P1, it is an RW node for partition P1. If it also manages the secondary replica of partition P4, it is also an RO node for partition P4.

[0079] RW nodes manage the primary replica of the partition, primarily responsible for handling data update transactions. They can perform write operations and generate commit logs (CLog, a log instantiated based on the WAL mechanism). RO nodes manage the secondary replicas of the partition, primarily providing read-only query services. They obtain data changes from the shared storage layer through synchronization mechanisms (e.g., distributed consensus protocols) to ensure data consistency, thereby achieving read-write separation, distributing the read load of RW compute nodes, and providing multi-read, multi-write capabilities.

[0080] Specifically, the commit log is a core component ensuring transaction durability and data consistency. It records all data operations performed during the transaction commit process, including but not limited to the transaction operation type, data changes, and timestamp information. The commit log is a crucial basis for fault recovery and data synchronization in distributed database systems. In shared storage mode, the working mechanism of the commit log differs from the traditional SN architecture. When executing a transaction, the RW node follows a log-first principle, pre-writing the data changes involved in the transaction to the commit log, and then updating the MEMTable based on the transaction execution. In this way, the commit log can record the generation process of incremental data in the MEMTable, and any node can recover data by replaying the commit log. The commit log generated by the RW node can then be uploaded to the shared storage layer (e.g., through a log service implemented based on the Multi-Paxos protocol) for archiving, allowing other computing nodes to pull the archived commit log from the shared storage layer for data replay, thereby maintaining data consistency between nodes.

[0081] In terms of data persistence, based on the storage characteristics of LSM-Tree, the SSTables in the shared storage layer adopt a multi-level organization method (as mentioned above, three levels: L0, L1, and L2). The generation and merging of SSTables at each level are achieved through different types of compaction operations.

[0082] See Figure 3When an RW node processes write transactions for primary replica partition data, it first writes the data to a MEMTable in memory. When the memory usage of the MEMTable reaches a preset threshold, to avoid blocking subsequent write operations, the RW node can freeze the MEMTable as an immutable Immutable MEMTable. Then, through a merge operation (referred to as a mini compaction to distinguish it from other types of merge operations), the data in the Immutable MEMTable is persisted to an SSTable file (referred to as a mini SSTable to distinguish it from SSTables generated by other types of merge operations). The execution of a mini compaction signifies that the incremental data in memory is officially written to disk; a mini compaction can also be called an incremental merge. After the mini compaction is completed, the mini SSTable is stored locally on the RW node. A background process can upload / write this mini SSTable to the L0 level of the shared storage layer, completing the persistence of the partition replica data in the MEMTable. As mentioned earlier, the LSM-Tree database system uses an append-only approach for data updates; therefore, the data in the mini SSTable can also be considered incremental data for the partition replicas.

[0083] When the number of mini SSTable files at the L0 level of the shared storage layer accumulates to a preset threshold, or when file fragmentation affects data query efficiency to a certain extent, a merge operation (referred to as minor compaction to distinguish it from other types of merge operations) can be performed to merge multiple mini SSTables. More specifically, minor compaction can be divided into two scenarios: one is merging multiple fragmented miniSSTables at the L0 level into a larger mini SSTable with stronger data continuity. Figure 3 (Not shown in the image) to reduce the number of files at the L0 level; another method is to merge multiple mini SSTables in the L0 level with the SSTable file in the L1 level (to distinguish it from other types of SSTables, this SSTable is called a minor SSTable) to generate a new minor SSTable at the L1 level, in order to reduce the overhead of cross-level queries.

[0084] In addition, the database system can periodically initiate a full merge operation (called a major compaction) to merge the SSTables at each level in the shared storage layer (in some practices, dynamic MEMTable data may also be included) into a single SSTable with the same version (called a major SSTable) as a new data baseline, thereby cleaning up expired data and optimizing the storage layout.

[0085] In the shared storage architecture described above, the RW node itself holds a complete data view of the partition, namely the sum of the incremental data in the current MEMTable and the data in the persistent SSTable in the shared storage. This allows the RW node to handle real-time read and write requests for the partition. The RO node, on the other hand, relies on the SSTable files in the shared storage and asynchronously synchronized commit logs to replay the incremental data of the partition, thus building a near real-time read-only data view. In other words, the RO node can pull the commit logs corresponding to the partition from the shared storage layer (or log service layer) and replay them locally, thereby updating the data state of the partition. This mechanism ensures that the RO node holds the real-time data of the partition without bearing the partition's write load, providing low-latency query services for the partition.

[0086] Therefore, in a distributed database system using shared storage, when a compute node (hereinafter referred to as the target node) fails and needs to be recovered, or when the database system needs to be elastically expanded by adding a new compute node (hereinafter referred to as the target node), the target node cannot directly rely on the shared storage layer to obtain a complete data view of the partition. This is because the shared storage layer only persists the data of the partition up to the last mini compaction. The MEMTable of the partition's RW node and the Immutable MEMTable that has not yet completed mini compaction still contain incremental data that has not been persisted to disk. This incremental data cannot be obtained by the target node from the SSTable file of the shared storage layer. Therefore, the target node needs to supplement this incremental data through log replay in order to form a complete data view of the partition.

[0087] Therefore, for the target node, the time point when the partition's RW node executes the mini compaction can be considered the starting point for incremental data replay (hereinafter referred to as the checkpoint). The inventors discovered through research that in related technologies, the execution time of the mini compaction is often used as the replay starting point for the target node; that is, the target node needs to replay all MEMTable data since the last mini compaction of the RW node. This means that the amount of data replayed by the target node directly depends on the execution frequency of the mini compaction. If the execution interval of the mini compaction is long, the amount of unpersisted data accumulated in the MEMTable will be large, leading to a long replay process and high I / O overhead, affecting the database system's Recovery Time Objective (RTO) and elastic scaling efficiency. Especially in TP scenarios with high concurrency access, a large amount of incremental data may accumulate rapidly in the MEMTable. If the target node's replay speed is low, it may cause database system service delays or interruptions, reducing the database system's sustainable service capabilities.

[0088] Since database systems also periodically perform full compaction operations (i.e., major compaction as mentioned above), the inventors further investigated the full compaction mechanism of distributed database systems under shared storage mode. They found that in related technologies, full compaction is not optimized for shared storage mode, and the operation still heavily relies on compute nodes. As a typical computationally and I / O-intensive operation, if full compaction is performed entirely on the compute nodes handling foreground tasks, it will compete with online read / write transactions for CPU time, memory, and network resources. Especially during peak business periods, this can easily lead to increased latency and decreased throughput in foreground transactions, and even write blocking, affecting the service stability of the database system in TP scenarios. Furthermore, because multiple compute nodes share the same set of data in shared storage mode, the lack of a coordinated full compaction mechanism for shared data in related technologies makes it easy for multiple compute nodes to simultaneously trigger write operations on the major SSTable, causing deadlocks and disrupting the merge sorting results of the major SSTable, thus limiting the executability of full compaction operations in distributed database systems under shared storage mode.

[0089] In view of this, the inventors have optimized the incremental and full merging methods for distributed database systems using a shared storage model in the embodiments of this specification. This can accelerate the generation of checkpoints, increase the frequency of incremental data being written to disk in MEMTable, thereby improving the continuous service capability of the database system during fault recovery or elastic expansion, and reducing the overhead of fault recovery or elastic expansion of the database system. Furthermore, by taking advantage of the characteristics of the shared storage architecture, the full merging can be offloaded from the compute node to the data node, reducing the occupation of compute node resources. Figure 4A and Figure 4B The implementation frameworks of the incremental merging optimization method and the full merging optimization method are shown respectively.

[0090] See Figure 4ATaking any one of the computing nodes in a distributed database system as an example, as time goes on, the incremental data written to the MEMTable increases continuously. As mentioned earlier, the commit log (CLog) recording the incremental data generation process also increases. Before freezing the MEMTable, several merge operations (this type of merge operation can be called microcompaction) can be performed on the incremental data in the active MEMTable. This persists the incremental data within a specific time period or a specific data volume range in memory into a data file SSTable. Correspondingly, the data file generated by performing this merge operation can be called a micro SSTable. The microSSTable can serve as a unit for persisting incremental data in the database system, containing a portion of the incremental data in the MEMTable. Subsequently, the commit log corresponding to the incremental data contained in the micro SSTable, recording its generation process, can be uploaded to the data node for archiving. After receiving the commit log sent by the computing node, the data node can update the checkpoint corresponding to the partition replica managed by that computing node. As mentioned earlier, checkpoints can serve as the starting point for data replay in scenarios such as fault recovery and elastic scaling. For example, the incremental data in MEMTable corresponds to the first replica. When the second compute node (which can be a newly added node or a node performing fault recovery) needs to rebuild the incremental data content of the first replica, it can obtain the corresponding commit log based on the first checkpoint corresponding to the first replica (generated based on the merge operation performed by the first compute node). This commit log records the generation process of the incremental data of the first replica in MEMTable after the first checkpoint. The compute node can replay the commit log and recover all incremental data that has not been persisted to SSTable after the first checkpoint in its MEMTable, thereby avoiding replaying the data in the complete MEMTable of the first compute node, significantly reducing the amount of log data to be replayed and the replay time. Increasing the update frequency of the checkpoint to be synchronized with the finer-grained micro compaction allows the data replay starting point during fault recovery or elastic scaling to be advanced, effectively solving the problem of excessive replay data volume caused by long mini compaction intervals. In addition, it should be understood that the checkpoints generated in the above embodiments can be stored in the data nodes of the shared storage layer, or in layers such as the metadata service layer or the log service layer. This specification does not limit this.

[0091] The above is about Figure 4A A brief introduction to the incremental merging optimization method is given below. Next, please refer to... Figure 4BIn the full-scale merge optimization method, the root service (RS) of the database system can initiate a full-scale merge task under preset triggering conditions (e.g., when the number of mini-compaction executions reaches a threshold, or during off-peak business periods). The first compute node among several compute nodes in the database system, responding to the full-scale merge task initiated by the root service, generates a first merge task for several first partitions it manages. The first compute node is the primary replica management node for these first partitions. The first compute node can write the generated first merge task to the metadata service shared by all nodes in the database system. Subsequently, data nodes in the shared storage layer can retrieve the first merge task by querying the metadata service and perform a full-scale merge (major compaction) on this task to obtain the target data file (i.e., a new majorSSTable, which can be used as the baseline data file). After completing the first merge task, the data nodes can update the execution status of the first merge task in the metadata service. Other compute nodes associated with the aforementioned first partitions (e.g., second compute nodes managing the primary or secondary replicas of these partitions) can monitor updates to the execution status in the metadata service and update the data in their local caches based on the generated target data files, thereby maintaining consistency between the compute node cached data and the baseline data in the shared storage layer. In this way, the resource-intensive full merge operation can be moved from the compute nodes responsible for handling online transactions to the data nodes, completely separating compute and storage resources and avoiding the problem of compute nodes competing for compute resources with foreground transactions while undertaking the full merge task. Simultaneously, data nodes can more efficiently access the persistent data files in their shared storage layer and utilize the internal bandwidth of the storage cluster for data transmission, improving the execution efficiency of the full merge operation and shortening the execution window. Task coordination and state synchronization through the metadata service achieve decoupling and asynchronous collaboration between compute nodes and data nodes, significantly improving the elasticity and sustainable service capabilities of the distributed database system in the shared storage mode.

[0092] Based on the above technical framework Figure 5 and Figure 6 Flowcharts of incremental merging and full merging methods in a database system according to embodiments of this specification are shown respectively. It is understood that the methods disclosed in the embodiments of this specification can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. The database system's storage architecture is a log structure merge tree (LSM-Tree), and the database system uses a shared storage architecture, including data nodes and several computing nodes.

[0093] See Figure 5In one embodiment, the method for incrementally merging a database system can be executed by a first computing node among several computing nodes, and includes at least the following steps: S501: Obtaining first incremental data and a first commit log from an active memory table MEMTable; the first commit log is used to record the generation process of the first incremental data. S503: Persisting the first incremental data to a local first data file by performing a first merge operation compaction. S505: Uploading the first commit log to the data node, causing the data node to update the memory checkpoint of the first computing node.

[0094] As previously mentioned, the storage architecture of the database system is a log structure merge tree (LSM-Tree). The data writing process, persistent storage method, and merging mechanism implemented in this architecture have been described earlier and will not be repeated here. The database system consists of several nodes and uses a shared storage architecture, as mentioned earlier, including data nodes and compute nodes. The following will describe in detail a method for incremental merging in a database system, with reference to the accompanying drawings and embodiments.

[0095] In step S501, the first incremental data and the first commit log are obtained from the active memory table MEMTable; the first commit log is used to record the generation process of the first incremental data.

[0096] The MEMTable, acting as a memory buffer in the LSM-Tree storage architecture, continuously receives and stores data write operations (e.g., insert, update, delete) from upper-layer applications. These operations are recorded in append-only format, forming incremental data. An active MEMTable refers to a MEMTable that is still receiving user transaction writes and has not been frozen as an ImmutableMEMTable. Its core function is to temporarily store the latest generated incremental data to support low-latency write responses. The first incremental data can be a portion of the incremental data in the MEMTable. The first commit log, generated based on the WAL mechanism, is a commit log corresponding to the first incremental data and records in detail the generation process of the first incremental data.

[0097] In a specific practice, the acquisition of the first incremental data and the first commit log is not triggered randomly, but can be acquired when preset conditions are met, in order to balance the frequency of incremental merging and system resource overhead. For example, it can be obtained as follows: under the condition that preset conditions are met, the first incremental data and the first commit log are acquired from the memory table that is in an active state; the preset conditions include: after the previous memory checkpoint update, the data newly written to the memory table reaches a first threshold, and / or, a first time interval has elapsed since the previous memory checkpoint update.

[0098] In this practice, the first threshold can be the size of the incremental data (e.g., 100MB) or the number of records in the incremental data (e.g., 10,000 records), and the first duration can be a time interval (e.g., 5 minutes). In an engineering implementation, the database system can maintain checkpoint metadata, recording the timestamp information of the last persistent operation performed on the in-memory table. When preset conditions are met, the system traces back from the current write pointer position of the in-memory table, extracts the newly added data content since the last checkpoint as the first incremental data, and synchronously extracts the corresponding commit log as the first commit log.

[0099] Next, in step S503: by performing the first merging operation compaction, the first incremental data is persisted to the local first data file.

[0100] As mentioned earlier, to distinguish it from mini compaction, minor compaction, and major compaction, the first merge operation is referred to as micro compaction in the embodiments of this specification. Specifically, the first incremental data can be merged, sorted, and compressed, and then persisted as an ordered SSTable data file (i.e., the first data file, also known as the micro SSTable). Since micro compaction only processes a portion of the incremental data in the MEMTable, its execution overhead is far lower than other types of merge operations. It can usually be completed in a very short time without blocking foreground operations, thus enabling rapid persistence of incremental data in the memory table to disk. In practice, this step can be executed through an asynchronous task.

[0101] After the first data file is generated, an asynchronous thread can be used to upload it to the data node. Furthermore, after uploading, the first data file can be deleted from the local persistent storage on the first compute node to free up local storage space. This post-upload deletion strategy reduces storage pressure on the compute node and ensures unified management of incremental data in shared storage, aligning with the stateless design principle of compute nodes. Deletion is typically performed after confirming successful receipt of the data file by the data node to avoid the risk of data loss.

[0102] After the first incremental data is written to disk, in step S505: the first commit log can be uploaded to the data node, so that the data node updates the memory checkpoint of the first computing node.

[0103] The first commit log records the generation process of the first incremental data and synchronizes it to the shared storage layer to support cross-node data synchronization and fault recovery. The upload process is typically executed asynchronously. After receiving the first commit log, the data node can update the memory checkpoint of the first compute node. In practice, the checkpoint can be metadata that records the location information of the currently persisted incremental data on the compute node, such as the Log Sequence Number (LSN) or data timestamp. After the update, the checkpoint indicates that the first incremental data has been safely persisted to disk. If subsequent recovery of the memory table is required, it is only necessary to replay the commit logs after this checkpoint.

[0104] By following the steps above, the frequency of incremental data being written to disk in the memory table can be increased from the mini compaction level to the finer-grained micro compaction level, significantly accelerating the generation of checkpoints and thus reducing the amount of commit log data that needs to be replayed during fault recovery or elastic scaling.

[0105] In a distributed database system scenario, multiple replicas of the partition data corresponding to the first incremental data in the MEMTable can be distributed and managed by multiple compute nodes. In this case, the first compute node can be the primary replica management node corresponding to the first incremental data, and its MEMTable stores the latest incremental data for the partition. Based on the characteristics of compute nodes in a distributed database system described above, the first compute node can also be a secondary replica management node for other partition data (hereinafter referred to as the second partition). Therefore, the first compute node also needs to synchronize with the compute node managing the primary replica of the second partition (hereinafter referred to as the second compute node) to obtain the incremental data of the second partition generated by the second compute node, in order to maintain the data consistency of the secondary replicas of the second partition it manages.

[0106] Specifically, the first compute node can receive a third commit log from the data node. This third commit log records the generation process of third incremental data, which is the incremental data of the second compute node among the plurality of compute nodes. The second compute node is the primary replica management node corresponding to the third incremental data, and the first compute node is the secondary replica management node corresponding to the third incremental data. The first compute node can replay the third commit log to store the third incremental data in the memory table. Taking partition data P2 as an example, the first compute node is the secondary replica management node of P2, and the second compute node is the primary replica management node of P2. The data nodes of the database system can act as the central hub for commit logs, storing the commit logs uploaded by each compute node (RW node). The first compute node can obtain the third commit log generated by the second compute node, which records the generation process of the P2 incremental data. The first compute node can replay the third commit log to replay the P2 incremental data into its MEMTable, thus completing the synchronization of the P2 data.

[0107] The above is an explanation of the incremental merging method in one embodiment. It should be noted that, for an active memory table, the above steps S501-S505 can be repeatedly triggered and executed over time to achieve continuous disk persistence of dynamic incremental data in the MEMTable and continuously advance the generation / update of checkpoints.

[0108] In practice, when the memory occupied by incremental data in the MEMTable reaches a certain threshold, minicompaction can be executed to write all incremental data in the MEMTable to disk. Specifically, the memory table of the first compute node can be frozen, changing it from a writable active state to a read-only immutable state, forming an Immutable MEMTable. To ensure that this freezing process does not affect the continuity of the foreground write service, a new active memory table can be created simultaneously with freezing the memory table to handle subsequent incremental data writes.

[0109] After the memory table is frozen as an immutable memory table, the second incremental data in the immutable memory table and the corresponding second commit log can be obtained. The second incremental data refers to all the incremental data content stored in the immutable memory table, representing all the incremental data accumulated by the memory table since the last mini compaction (or initial state); the second commit log is the commit log corresponding to these incremental data, which fully records the generation process of the second incremental data.

[0110] Next, a second merge operation can be performed to persist all incremental data in the immutable memory table to a local second data file based on the second incremental data. Here, the second merge operation refers to minicompaction, and the second data file refers to a mini SSTable. In a specific practice, the previously generated micro SSTable can be used to accelerate the mini compaction process. In this practice, the second incremental data can be the difference between all incremental data in the immutable memory table and several first incremental data sets. As mentioned earlier, the first incremental data are the incremental data of the memory table that was persisted to disk during the previous micro compaction. That is, minicompaction only needs to process the incremental data that has been added since the last micro compaction and has not been persisted, thereby avoiding repeated processing of already persisted data and significantly accelerating the execution of mini compaction. When performing the second merge operation, all incremental data in the immutable memory table can be persisted to a second data file based on the second incremental data and the several first incremental data sets. For example, several microSSTable data files corresponding to the several first incremental data sets can be directly referenced, and the difference incremental data (i.e., the second incremental data) can be merged to generate the second data file.

[0111] After the second data file is generated, it and the second commit log can be uploaded to the data node via an asynchronous thread. Upon receiving the second commit log, the data node can update the memory checkpoint of the first compute node based on it. After uploading the second data file to the data node, it can also be deleted from the local persistent media on the first compute node to free up local storage space.

[0112] By following the steps above, incremental merging of active-state memory tables can be achieved in distributed database systems employing shared storage architecture, thus advancing the generation / updating of memory checkpoints. In data recovery or elastic scaling scenarios, when the first compute node needs to rebuild a replica (hereinafter referred to as the first replica), the memory checkpoints generated / updated in the aforementioned steps can be used to accelerate data reconstruction.

[0113] Taking the first computing node among several computing nodes of the database system as an example, when the first computing node is used to perform the reconstruction operation of the first replica, it can first obtain the memory checkpoint corresponding to the first replica from the data node. The memory checkpoint is generated by the method described above and records the relevant information of the last persistence of the incremental data of the first replica. It can be used to indicate the data starting point for replaying the incremental data of the first replica in the memory table MEMTable.

[0114] Next, the first compute node can obtain the first commit log from the data node. The first commit log is used to record the generation process of the first incremental data, which is the incremental data generated by the first replica after the memory checkpoint;

[0115] Finally, the first compute node can store the first incremental data into the memory table by replaying the first commit log. Specifically, the first compute node can parse the first commit log line by line, re-execute the data operation instructions recorded therein, and regenerate the incremental data corresponding to these operation instructions in the first compute node's MEMTable.

[0116] It is understandable that when the first compute node performs a rebuild operation on the first replica in response to elastic scaling, the rebuild operation can be replica creation. The newly added compute node only needs to replay the newly generated incremental data since the memory checkpoint, without having to replay the entire MEMTable, which greatly shortens the replica's online time. When the first compute node performs a rebuild operation on the first replica in response to fault recovery, the rebuild operation can be replica recovery. After restarting or being replaced, the first compute node needs to quickly restore the data state before the fault. Due to the rapid forward advancement of the memory checkpoint, the amount of logs that the first compute node needs to replay can be controlled within a small range, which significantly reduces the recovery time and improves the availability of the database system.

[0117] The above describes the incremental merging method disclosed in the embodiments of this specification. In another embodiment, for a distributed database system using a shared storage architecture, a full merge can be performed using the following method. See also... Figure 6 The method for performing a full merge on a data system includes at least the following steps: S601: A first computing node among several computing nodes responds to a full merge task initiated by the root service of the database system and generates a first merge task for several first partitions; the first computing node is the primary replica management node for the several first partitions. S603: The first computing node writes the first merge task into a metadata service shared by all nodes. S605: The data node obtains the first merge task through the metadata service and performs a full merge compaction on the first merge task to obtain a target data file. S607: The data node updates the execution status of the first merge task in the metadata service. S609: A second computing node associated with the several first partitions responds to the update of the execution status and updates the relevant cached data according to the target data file.

[0118] As previously mentioned, the storage architecture of the database system is a log structure merge tree (LSM-Tree). The data writing process, persistent storage method, and merging mechanism implemented in this architecture have been described earlier and will not be repeated here. The database system consists of several nodes and uses a shared storage architecture, as mentioned earlier, including data nodes and compute nodes. The following will describe in detail a method for performing a full data merge in the database system, with reference to the accompanying drawings and embodiments.

[0119] Step S601: The first computing node among the plurality of computing nodes responds to the full merge task initiated by the root service of the database system and generates a first merge task for the plurality of first partitions; the first computing node is the primary replica management node of the plurality of first partitions.

[0120] The root service is the coordination component in a distributed database system, responsible for global resource management, task scheduling, and system metadata maintenance. It is typically deployed in a highly available manner. The root service can initiate a major compaction task based on predefined strategies (e.g., scheduled triggering, system load conditions, etc.), such as automatically triggering it during off-peak business periods to minimize the impact on online services.

[0121] The first compute node, acting as the primary replica management node for the plurality of first partitions, has write permissions to write data related to the plurality of first partitions to the shared storage layer. Upon receiving a full merge task initiated by the root service, the first compute node can generate subtasks for the plurality of first partitions. In this embodiment, a first merge task involving a full merge of the plurality of first partitions is used as an example. However, it should be understood that in practice, the first compute node can freely combine the various partitions within the plurality of first partitions to generate several subtasks as needed.

[0122] After the first merge task is generated, in order to distribute the first merge task to the data nodes, in step S603, the first computing node writes the first merge task into the metadata service shared by each node.

[0123] As mentioned earlier, metadata service is one of the core components of a shared storage architecture. As a highly available, persistent storage service deployed independently or integrated within the shared storage layer, it can be used to store and manage global metadata of the database system, including but not limited to metadata about data partitions (e.g., version, file list, location mapping), task queues and status, system configuration, etc. Both compute nodes and data nodes in the database system can access the metadata service through standard interfaces, thereby achieving centralized management and cross-node sharing of metadata.

[0124] The first computing node publishes the first merging task by writing it into the metadata service, thereby decoupling task generation from task execution. This allows any node with access to the metadata service to discover and obtain the first merging task.

[0125] In step S605, the data node can obtain the first merging task through the metadata service, and perform a full merging compaction on the first merging task to obtain the target data file.

[0126] Data nodes in the shared storage layer hold SSTable data files of the database system and possess computational and I / O capabilities, enabling them to perform resource-intensive tasks. Data nodes can periodically poll or obtain the first merge task to be processed from the metadata service via an event-driven mechanism.

[0127] After receiving the first merge task, the data node can perform a full merge, reading the SSTable data files of the relevant partitions (i.e., the aforementioned first partitions), performing operations such as sorting, deduplication, and compression, to generate a new, version-consistent target data file (i.e., the major SSTable). In practice, the SSTable data files used to perform the full merge may include:

[0128] Incremental data files: namely micro SSTable, mini SSTable, or minor SSTable. In some implementations, they may also be incremental data in MEMTable.

[0129] Baseline data file: This is the major SSTable data file generated through the preceding major compaction.

[0130] After obtaining the data file used for performing a full merge, the data node can perform the full merge operation locally. This operation is a computationally and I / O-intensive process. The data node needs to merge and sort multiple SSTable data files at different levels, remove duplicate data items based on the key (i.e., keep the latest version), clean up data records that have been marked for deletion or expired, and reorganize and compress all valid data to generate a target data file, i.e., a new major SSTable, which represents a complete and non-redundant data baseline of the several first partitions at the time point of the first merge task.

[0131] It is understandable that, in order to free up storage space in a timely manner, the various data files used to perform the full merge can be deleted after the target data files are generated. Furthermore, it should be noted that the data nodes used to perform the full merge for the first merge task can be execution nodes selected by the database system from among several data nodes, for example, data nodes selected based on a load balancing mechanism.

[0132] Through these steps, the resource-intensive full merge operation can be completely offloaded from the compute nodes responsible for handling online transactions to data nodes dedicated to data file storage. Data nodes typically possess optimized local I / O capabilities and high-bandwidth access to SSTable data files, enabling efficient reading, sorting, and writing of large-scale data. This avoids resource contention with foreground transactions that would arise from performing a full merge on the compute nodes, thus ensuring the stability and performance of the database system's online services.

[0133] Next, in step S607, the data node can update the execution status of the first merging task in the metadata service.

[0134] After successfully completing the full merge operation of the first merge task and persisting the target data file, the execution status of the first merge task in the metadata service can be updated, for example, from "in progress" or "pending" to "completed". In practice, along with updating the status of the merge task, key metadata related to the full merge result is usually also recorded, such as the unique identifier of the target data file, storage path, data version number, and file size. This metadata can be recorded in the metadata service and associated with the metadata of the corresponding partition (i.e., the several first partitions), thereby switching the data baseline of the partition to the target data file.

[0135] In some practices, the root service can also perform a first data verification based on the target data file to ensure the integrity and consistency of the data after full merging. This first data verification includes one or more of the following verification mechanisms: data consistency verification of the primary table and its corresponding index table, and data consistency verification between partition replicas. Data verification between the primary table and the index table ensures the consistency of data between them. For example, it can verify whether the checksum of the index column in the primary table matches the checksum of the corresponding column in the index table. Data verification between partition replicas focuses on verifying cross-node data consistency. The root service can coordinate multiple compute nodes managing the same partition to load the data of that partition from the target data file and calculate the checksum locally. The root service can then compare the checksums returned by each compute node. If they are the same, it proves that each compute node can correctly obtain the data of that partition based on the target data file.

[0136] Next, in step S609, the second computing node associated with the plurality of first partitions can update the relevant cache data according to the target data file in response to the update of the execution state.

[0137] The second compute node refers to the compute node associated with the plurality of first partitions, including but not limited to read-only compute nodes (RO nodes) that manage replicas of any of the first partitions, and read-write compute nodes (RW nodes) that manage primary replicas of any of the first partitions. The second compute node typically maintains a local data cache, which caches hot data in the partitions managed by the second compute node to accelerate the reading of frequently accessed data, reduce I / O dependence on the shared storage layer, and improve query performance.

[0138] When the second compute node detects that the execution status of the first merge task in the metadata service has been updated to "completed," it can trigger the cached data update process. Specifically, the second compute node can periodically poll the execution status of the merge tasks stored in the metadata service, or detect updates to the execution status of the merge tasks in real time through an event listening mechanism. When the second compute node detects a change in the execution status of the first merge task, it can determine whether the first merge task has been successfully completed and whether the target data file is readable based on the metadata of the first merge task.

[0139] If the first merge task is completed and the target data file is readable, the second compute node can determine several hot data rows from the partition it manages (hereinafter referred to as the target partition) based on historical query statistics. To improve the accuracy of identifying hot data rows, the second compute node can use either the Least Recently Used (LRU) or Least Frequently Used (LFU) algorithm to determine hot data rows. The LRU algorithm focuses on the access timeliness of data rows, filtering out frequently queried datasets by recording the most recent access timestamps of data rows in the target partition. For example, the top 20% of data rows accessed in the past hour are identified as hot data rows. The LFU algorithm focuses on the access frequency of data rows, identifying data rows with access counts exceeding a preset threshold as hot data rows by statistically analyzing the cumulative access counts of data rows. In practical applications, the second compute node can also use an Adaptive Replacement Cache (ARC) algorithm to dynamically adjust the weight ratio of LRU and LFU to adapt to the workload characteristics of the database system. For example, in TP scenarios with obvious time patterns, the LRU weight can be appropriately increased, and in AP scenarios with many repeated queries, the LFU weight can be appropriately increased to ensure that the identified hot data rows can cover the actual data access needs on a larger scale and improve the cache hit rate.

[0140] After identifying the hot data rows, the second computing node can read the hot data rows based on the target data file in the shared storage layer.

[0141] To ensure the correctness of cached data and prevent errors during data transmission or loading, after successfully loading hot data rows into the local cache according to the above steps, the second compute node can perform a second data verification based on the target data file to ensure data integrity. This verification process typically uses the Cyclic Redundancy Check (CRC) algorithm to calculate the checksum of the cached hot data rows in the second compute node and compare it with the checksum of these hot data rows in the target data file. If the checksums do not match, the second compute node can automatically trigger data repair, rereading the data rows from the target data file. This process can be repeated until the second data verification passes, or until the maximum number of retries is reached before reporting an error. This mechanism ensures the consistency between cached data and persistent data, making the data query service provided by the second compute node based on the cache completely consistent with the results of directly accessing the target data file, thus improving the service reliability of the database system under the shared storage architecture.

[0142] While the above methods can utilize data nodes to perform full merge operations, avoiding the competition for computing resources between compute nodes and foreground transactions, uneven distribution of computing resources may still exist in practical applications. On one hand, if data nodes in the shared storage layer simultaneously handle full merge tasks for multiple tenants or partitions, the efficiency of the full merge may decrease due to concentrated I / O and computation, potentially even affecting the stability of the shared storage layer's data read / write services. On the other hand, in distributed clusters of database systems, some compute nodes are often under low load (e.g., compute nodes not fully utilized after elastic scaling, or idle RO nodes during off-peak periods), and their computing resources are not effectively utilized. Since the execution of a full merge operation involves relatively heavy computational work, and compute nodes typically have stronger computing resource configurations than data nodes, to further optimize resource scheduling efficiency for full merge operations and fully utilize idle cluster computing resources, the computational work of the first merge task can be assigned to compute nodes with low computational load in the cluster. This achieves dynamic balancing of computing resources and improves the execution efficiency of the full merge. The following will provide a detailed description of the execution method for full merging, with reference to specific examples.

[0143] The database system can monitor the computing load of each computing node, including but not limited to CPU utilization, memory usage, and I / O load. When the computing load of a computing node (hereinafter referred to as the third computing node) is found to be below a preset threshold, the third computing node can be included in the low-load resource pool as a candidate node for executing the full merge task.

[0144] The first computing node can create a first log stream (LS) based on the first merge task. The first log stream is the logical carrier of data and task information for the first merge task, and may contain information such as the identification information of the several first partitions and a list of data files to be merged.

[0145] The first computing node can grant write permissions to the third node for the first log stream. The write permissions apply to the data files stored in the data nodes that correspond to the plurality of first partitions, ensuring that within a specific time window (e.g., within the validity period of the write permissions), only the third node with the write permissions has the right to modify the aforementioned data files corresponding to the plurality of first partitions.

[0146] The third compute node, based on the first log stream, retrieves data from the data nodes and performs a full merge to generate the first data. After obtaining authorization, the third compute node first loads the first log stream, parses the first merge task, and retrieves the SSTable data files to be merged from the data nodes in the shared storage layer. These data files may be distributed across different storage layers, including micro or mini SSTables at layer L0, minor SSTables at layer L1, and major SSTables (historical baseline data files) at layer L2. The third compute node performs a full merge operation on the retrieved data files to generate the first data. In this way, the computation for the full merge operation is completed locally on the third compute node, fully utilizing its idle computing resources without interfering with the data storage services of the data nodes.

[0147] Then, the third compute node can write the first data to the target data file of the data node. After the full merge computation for the first merge task is completed, the third compute node can upload the newly generated first data to the target location of the data node. After the upload is complete, the third compute node can update the execution status of the first merge task to the metadata service and release the previously acquired write permissions. After receiving the first data, the data node can update the relevant metadata, so that the newly generated / updated target data file can be accessed by other nodes in the cluster as baseline data.

[0148] It should be noted that in practical applications, the two full merge execution methods can be dynamically switched based on the actual load of each node in the cluster (including compute nodes and data nodes). When the data node load is low, the full merge can be performed by the data node first; when the data node load is high or there are low-load compute nodes in the cluster, the full merge can be performed by the compute nodes to make full use of idle computing resources.

[0149] The foregoing, based on one or more embodiments, details a method for incremental merging and full merging in a database system. Using the methods provided in the embodiments of this specification, in a distributed database system employing a shared storage architecture, it can accelerate the generation of checkpoints, increase the frequency of incremental data writes to disk in the MEMTable, thereby improving the continuous service capability of the database system during fault recovery or elastic scaling, and reducing overhead. Furthermore, leveraging the characteristics of the shared storage architecture, it can move the resource-intensive full merging operation from the computing nodes responsible for handling online transactions to the data nodes, completely separating computing and storage resources. This avoids the problem of computing nodes competing for computing resources with foreground transactions due to the execution of full merging tasks. Through metadata services for task coordination and state synchronization, it achieves decoupling and asynchronous collaboration between computing nodes and data nodes, significantly improving the elasticity and sustainable service capability of the distributed database system under a shared storage architecture.

[0150] In this specification, the terms "first" in the first computing node, the first incremental data, etc., and the corresponding "second" and "third" (if they exist) in the text are merely for the convenience of distinction and description, and do not have any limiting meaning.

[0151] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0152] Figure 7 This is a schematic diagram of an apparatus for incremental merging in a database system according to an embodiment of this specification. The apparatus 700 is deployed in a computing device, which can be implemented using any device, equipment, platform, or device cluster with computing and processing capabilities. The database system's storage architecture is a log structure merge tree (LSM-Tree), and the database system includes data nodes and several computing nodes; the apparatus is deployed on the first computing node among the several computing nodes. This apparatus embodiment is similar to... Figure 5 Corresponding to the method embodiment shown, the apparatus 700 includes:

[0153] The acquisition module 701 is configured to acquire first incremental data and first commit log from the active memory table MEMTable; the first commit log is used to record the generation process of the first incremental data.

[0154] The merging module 702 is configured to persist the first incremental data to a local first data file by executing a first merging operation compaction.

[0155] The upload module 703 is configured to upload the first commit log to the data node, so that the data node updates the memory checkpoint of the first computing node.

[0156] Figure 8 This is a schematic diagram of an apparatus for incremental merging in a database system according to an embodiment of this specification. The apparatus 800 is deployed in a computing device, which can be implemented using any device, equipment, platform, or device cluster with computing and processing capabilities. The database system has a log structure merge tree (LSM-Tree) storage architecture and includes data nodes and several computing nodes. The apparatus is deployed on a first computing node among the several computing nodes, and the first computing node is used to perform a reconstruction operation of a first replica. The apparatus 800 includes:

[0157] The acquisition module 801 is configured to acquire the memory checkpoint corresponding to the first replica from the data node. The memory checkpoint is generated using the device of claim 12 and is used to indicate the data starting point for replaying incremental data of the first replica in the memory table MEMTable.

[0158] The acquisition module 801 is further configured to acquire a first commit log from the data node; the first commit log is used to record the generation process of the first incremental data, and the first incremental data is the incremental data generated by the first replica after the memory checkpoint.

[0159] The replay module 802 is configured to replay the first commit log in order to store the first incremental data into the memory table.

[0160] Figure 9 This is a schematic diagram of an apparatus for performing a full data merge in a database system according to an embodiment of this specification. The apparatus 900 is deployed in a computing device, which can be implemented using any device, equipment, platform, or device cluster with computing and processing capabilities. The database system's storage architecture is a log structure merge tree (LSM-Tree), and the database system includes several data nodes and several computing nodes; the apparatus is deployed on the first data node among the several data nodes. The apparatus 900 includes:

[0161] The acquisition module 901 is configured to acquire a first merge task from the metadata service shared by each node. The first merge task is generated by the first computing node and is used to instruct a full merge compaction to be performed on several partitions. The first computing node is the master replica management node of the several partitions.

[0162] The merging module 902 is configured to perform a full merge based on the data files corresponding to the partitions to obtain the target data file.

[0163] The update module 903 is configured to update the execution status of the first merge task in the metadata service, so that the computing nodes related to the plurality of partitions update the relevant cache data according to the target data file.

[0164] Figure 10 This is a schematic diagram of a system for performing a full data merge in a database system, according to an embodiment of this specification. The database system's storage architecture is a log structure merge tree (LSM-Tree), and the database system includes data nodes and several compute nodes. This system embodiment is similar to... Figure 6 Corresponding to the method embodiment shown, the system 1000 includes:

[0165] The first computing node 1001 is configured to generate a first merge task for several first partitions in response to a full merge task initiated by the root service of the database system; the first computing node is the primary replica management node of the several first partitions.

[0166] The first computing node 1001 is also configured to write the first merging task into a metadata service shared by all nodes.

[0167] Data node 1002 is configured to obtain the first merging task through the metadata service, and perform a full merging compaction on the first merging task to obtain the target data file.

[0168] The data node 1002 is also configured to update the execution status of the first merging task in the metadata service.

[0169] The second computing node 1003 is configured to update relevant cached data according to the target data file in response to the update of the execution status; the second computing node is associated with the plurality of first partitions.

[0170] According to another embodiment, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the foregoing combinations. Figure 5 or Figure 6 The steps of the method are described.

[0171] According to yet another embodiment, this specification also provides a computing device including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the foregoing combination. Figure 5 or Figure 6 The steps of the method are described.

[0172] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0173] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for incremental merge in a database system, a storage architecture of the database system being a Log-Structured Merge Tree (LSM-Tree), the database system comprising a data node and a number of compute nodes. The method is executed by the first computing node among the plurality of computing nodes, including: The first incremental data and the first commit log are obtained from the active memory table MEMTable; the first commit log is used to record the generation process of the first incremental data. By performing the first merging operation compaction, the first incremental data is persisted to the local first data file; The first commit log is uploaded to the data node, causing the data node to update the memory checkpoint of the first compute node.

2. The method according to claim 1, wherein, The step of retrieving the first incremental data and the first commit log from the active memory table includes: Under certain preset conditions, the first incremental data and the first commit log are obtained from the active memory table; the preset conditions include: after the previous memory checkpoint update, the data newly written to the memory table reaches a first threshold, and / or, a first time interval has elapsed since the previous memory checkpoint update.

3. The method according to claim 1, further comprising: Upload the first data file to the data node.

4. The method according to claim 3, wherein, After uploading the first data file to the data node, the method further includes: Delete the first data file from the local persistent media.

5. The method according to claim 1, further comprising: After the memory table is frozen as an immutable memory table, the second incremental data in the immutable memory table and the corresponding second commit log are obtained; The second commit log is used to record the generation process of the second incremental data; By performing a second merge operation, based on the second incremental data, all incremental data in the immutable memory table is persisted to a local second data file; Upload the second data file to the data node; The second commit log is uploaded to the data node, so that the data node updates the memory checkpoint of the first compute node based on the second commit log.

6. The method according to claim 5, wherein, The second incremental data is the difference between all the incremental data and a plurality of the first incremental data; By performing a second merge operation, based on the second incremental data, all incremental data in the immutable memory table is persisted to a second data file, including: By performing a second merge operation, based on the second incremental data and the plurality of first incremental data, all incremental data in the immutable memory table are persisted to a second data file.

7. The method according to claim 5, wherein, After uploading the second data file to the data node, the method further includes: Delete the second data file from the local persistent media.

8. The method according to claim 1, further comprising: Receive the third commit log from the data node; The third commit log is used to record the generation process of the third incremental data. The third incremental data is the incremental data of the second computing node among the plurality of computing nodes. The second computing node is the primary replica management node corresponding to the third incremental data. The third commit log is replayed to store the third incremental data into the memory table.

9. The method according to claim 1, wherein, The first computing node is the primary replica management node corresponding to the first incremental data.

10. A method for incremental merging in a database system, wherein the storage architecture of the database system is a log structure merge tree (LSM-Tree), and the database system includes data nodes and several computing nodes; The method is executed by a first computing node among the plurality of computing nodes, the first computing node being used to perform a reconstruction operation of the first replica; the method includes: Obtain the memory checkpoint corresponding to the first replica from the data node. The memory checkpoint is generated by the method of any one of claims 1-9 and is used to indicate the data starting point for replaying incremental data of the first replica into the memory table MEMTable. The first commit log is obtained from the data node; the first commit log is used to record the generation process of the first incremental data, which is the incremental data generated by the first replica after the memory checkpoint; Replay the first commit log to store the first incremental data into the memory table.

11. The method according to claim 10, wherein, The reconstruction operation is any one of the following: copy restoration, copy creation.

12. A method for performing a full merge in a database system, wherein the storage architecture of the database system is a log structure merge tree (LSM-Tree), and the database system includes several data nodes and several computing nodes; The method is executed by a first data node among the plurality of data nodes, including: The first merge task is obtained from the metadata service shared by each node. The first merge task is generated by the first compute node and is used to instruct a full merge compaction to be performed on several partitions. The first compute node is the master replica management node of the several partitions. Based on the data files corresponding to the partitions, a full merge is performed to obtain the target data file; The execution status of the first merge task in the metadata service is updated so that the computing nodes associated with the plurality of partitions update the relevant cache data according to the target data file.

13. The method according to claim 12, wherein, Based on the data files corresponding to the aforementioned partitions, a full merge is performed to obtain the target data file, including: Receive first data sent by the second computing node; the first data is generated by the second computing node performing a full merge based on the first log stream, the first log stream is generated based on the first merge task, and the computing load of the second computing node is lower than a preset threshold; Write the first data into the target data file.

14. The method according to claim 13, wherein, The second computing node has write permissions related to the first log stream granted by the first computing node, and the write permissions apply to the data files in the data node corresponding to the plurality of partitions.

15. The method according to claim 12, wherein, The data files include one or more of the following: incremental data files and baseline data files.

16. The method according to claim 12, wherein, After obtaining the target data file, the method further includes: Delete the aforementioned data files.

17. The method according to claim 12, wherein, After obtaining the target data file, the method further includes: In the metadata service, the metadata corresponding to each of the several partitions is updated.

18. The method according to claim 12, wherein, The first data node is the execution node selected by the metadata service from the plurality of data nodes for the first merging task.

19. A method for performing a full merge in a database system, wherein the database system has a storage architecture of a log structure merge tree (LSM-Tree) and includes data nodes and several compute nodes; the method includes: The first computing node among several computing nodes responds to the full merge task initiated by the root service of the database system and generates a first merge task for several first partitions; The first computing node is the primary replica management node of the plurality of first partitions; The first computing node writes the first merging task into the metadata service shared by all nodes; The data node obtains the first merging task through the metadata service, and performs a full merging compaction on the first merging task to obtain the target data file; The data node updates the execution status of the first merging task in the metadata service; In response to the update of the execution state, the second computing node associated with the plurality of first partitions updates the relevant cached data according to the target data file.

20. The method according to claim 19, wherein, After the data node updates the execution status of the first merging task in the metadata service, the method further includes: The root service performs a first data verification based on the target data file; the first data verification includes one or more of the following: data verification of the main table and its corresponding index table, and data verification between partition replicas.

21. The method according to claim 19, wherein, The second computing node is the replica management node for the target partition among the plurality of first partitions; the second computing node associated with the plurality of first partitions, in response to the update of the execution status, updates the relevant cached data according to the target data file, including: The second computing node reads the execution status and determines whether the first merging task has been completed; If the first merging task has been completed, the second computing node determines several hot data rows in the target partition; The second computing node reads the several hot data rows from the target data file and stores them in the local cache.

22. The method according to claim 21, wherein, The second computing node determines several hot data rows in the target partition, including: The Least Recently Used (LRU) algorithm and / or the Least Frequently Used (LFU) algorithm are used to identify several hot data rows from the target partition.

23. The method according to claim 21, wherein, After the second computing node reads the several hot data rows from the target data file and stores them in the local cache, the method further includes: The second computing node performs a second data verification on the target data file for the several hot data rows.

24. The method according to claim 19, wherein, For the first merge task, a full merge is performed to obtain the target data file, including: The first computing node grants write permissions to the third computing node for the first log stream, and the write permissions apply to the data files in the data node corresponding to the plurality of first partitions; the first log stream is created based on the first merge task, and the computing load of the third computing node is lower than a preset threshold; The third computing node obtains data from the data node based on the first log stream and performs a full merge to generate the first data; The third computing node writes the first data into the target data file of the data node.

25. An apparatus for incremental merging in a database system, wherein the storage architecture of the database system is a log structure merge tree (LSM-Tree), and the database system includes data nodes and several computing nodes; The device is deployed on the first computing node among the plurality of computing nodes, including: The acquisition module is configured to acquire first incremental data and a first commit log from an active memory table MEMTable; the first commit log is used to record the generation process of the first incremental data. The merging module is configured to persist the first incremental data to a local first data file by executing a first merging operation, compaction. The upload module is configured to upload the first commit log to the data node, so that the data node updates the memory checkpoint of the first computing node.

26. An apparatus for incremental merging in a database system, wherein the storage architecture of the database system is a log structure merge tree (LSM-Tree), and the database system includes data nodes and several computing nodes; The device is deployed on a first computing node among the plurality of computing nodes, the first computing node being used to perform a reconstruction operation of the first replica; the device includes: The acquisition module is configured to acquire the memory checkpoint corresponding to the first replica from the data node. The memory checkpoint is generated using the device of claim 12 and is used to indicate the data starting point for replaying incremental data of the first replica in the memory table MEMTable. The acquisition module is further configured to acquire a first commit log from the data node; the first commit log is used to record the generation process of the first incremental data, and the first incremental data is the incremental data generated by the first replica after the memory checkpoint; The replay module is configured to replay the first commit log in order to store the first incremental data into the memory table.

27. An apparatus for performing a full merge in a database system, the database system having a storage architecture of a log structure merge tree (LSM-Tree), the database system comprising a plurality of data nodes and a plurality of compute nodes; the apparatus being deployed on a first data node among the plurality of data nodes, comprising: The acquisition module is configured to acquire a first merge task from the metadata service shared by each node. The first merge task is generated by the first compute node and is used to instruct a full merge compaction to be performed on several partitions. The first compute node is the master replica management node of the several partitions. The merging module is configured to perform a full merge based on the data files corresponding to the partitions to obtain the target data file; The update module is configured to update the execution status of the first merge task in the metadata service, so that the computing nodes associated with the plurality of partitions update the relevant cache data according to the target data file.

28. A system for performing full data merging in a database system, wherein the database system has a storage architecture of a log structure merge tree (LSM-Tree), and the database system includes data nodes and several compute nodes; the system includes: The first computing node is configured to generate a first merge task for several first partitions in response to a full merge task initiated by the root service of the database system. The first computing node is the primary replica management node of the plurality of first partitions; The first computing node is further configured to write the first merging task into a metadata service shared by all nodes. The data node is configured to obtain the first merging task through the metadata service, and perform a full merging compaction on the first merging task to obtain the target data file; The data node is also configured to update the execution status of the first merging task in the metadata service; The second computing node is configured to update relevant cached data based on the target data file in response to an update of the execution status; the second computing node is associated with the plurality of first partitions.

29. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-24.

30. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-24.

Images