Distributed database incremental snapshot method and device and computer equipment

Abstract

The application discloses a distributed database incremental snapshot method and device and computer equipment. The method comprises the following steps: obtaining a change log from a distributed database node, wherein the change log comprises transaction key information; assembling the change log to obtain an assembly result; using a transaction start timestamp as a sorting key, performing merge sorting on the assembly result through a log structure merge tree structure to generate an ordered lake format file; using a Paimon interface to generate a snapshot ID reflecting a current database state based on a transaction start timestamp or a table time field; and querying the ordered lake format file and the snapshot ID by using a standard data lake reader. The method of the application can not only simplify the data migration process from the distributed database to the data lake, but also improve the efficiency and reliability of the whole process, and ensure the consistency and integrity of the data transmission between different systems.

Description

TECHNICAL FIELD

[0001] The present application relates to a database processing method, in particular to a distributed database incremental snapshot method, device and computer equipment. BACKGROUND

[0002] A distributed database is an innovative database system that provides higher scalability, availability and performance by distributing data across multiple physical nodes connected by a network. Compared with traditional centralized databases, this approach not only improves data processing efficiency, but also enhances the fault tolerance of the system. In a distributed database, transactions are the core mechanism to ensure data consistency, usually based on the Percolator algorithm, which can effectively support transaction management in a large-scale data environment. On the other hand, the paimon data lake format, as an emerging data storage solution, combines the advantages of streaming and batch processing operations, and combines data lake format with LSM (Log-Structured Merge) structure, realizing the support for real-time data updates.

[0003] Currently, in application scenarios requiring high accuracy and high performance, distributed databases are widely used, especially in cases where data needs to be exported to a data lake for backup or cleaning. However, existing methods either rely on SQL to read data and write to a data lake, or import data into a data lake by reading the change log of a distributed database, which has limitations in terms of operation complexity and efficiency; and the existing technology realizes cross-node consistent snapshot through global transaction management and recording transaction state, which not only increases the complexity and resource consumption of the system, but also may cause performance bottlenecks and scalability problems, especially when handling high concurrency and large-scale data; at the same time, the data import scheme relying on a specific change log format limits flexibility and the recovery process is complex in the event of a failure, these factors together constitute the main challenges in realizing efficient, flexible and high-performance distributed database applications.

[0004] Therefore, it is necessary to design a new method to not only simplify the data migration process from a distributed database to a data lake, but also improve the efficiency and reliability of the entire process, and ensure the consistency and integrity of data transmission between different systems. SUMMARY

[0005] The present application aims to overcome the defects of the prior art and provide a distributed database incremental snapshot method, device and computer equipment.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solution: a distributed database incremental snapshot method, comprising:

[0007] Obtain a change log from a distributed database node, wherein the change log comprises transaction key information;

[0008] Transaction assembly is performed on the change log to obtain an assembly result;

[0009] The start timestamp of the transaction is used as a sorting key, and the assembly result is sorted by a log-structured merge tree structure to generate an ordered lake format file;

[0010] Based on the transaction start timestamp or the table internal time field, a snapshot ID reflecting the current database state is generated using the Paimon interface;

[0011] The ordered lake format file and the snapshot ID are queried using a standard data lake reader.

[0012] Further technical solutions thereof are that the change log obtained from the distributed database node comprises:

[0013] The change log obtained from the distributed database node is obtained through an RPC protocol.

[0014] Further technical solutions thereof are that the transaction key information comprises a unique identifier, a prewrite, a rollback or a committed log type, an operation type, a start timestamp, a commit timestamp, a modified value and a modified value.

[0015] Further technical solutions thereof are that the transaction assembly is performed on the change log to obtain an assembly result, comprising:

[0016] For each change log, a unique identifier and a start timestamp are extracted to generate a Mid value, and classification processing is performed according to the log type to obtain an assembly result.

[0017] Further technical solutions thereof are that the classification processing according to the log type comprises:

[0018] When the log type is prewrite, the change log is written into a cache; when the log type is rollback, the change log corresponding to the Mid value is deleted from the cache; and when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value to form a data list containing disordered change logs.

[0019] Further technical solutions thereof are that the start timestamp of the transaction is used as a sorting key, and the assembly result is sorted by a log-structured merge tree structure to generate an ordered lake format file, comprising:

[0020] The start timestamp of the distributed transaction is set as a sorting key;

[0021] The received disordered transaction change log is written into the 0 layer of the log-structured merge tree in the Paimon data lake format, and the start timestamp is explicitly specified as the sorting field;

[0022] When data is written into the 0 layer, a merge of the log-structured merge tree is triggered, and in the merging process, the assembly result is sorted according to the start timestamp, and the assembly result is reassembled into a file conforming to the data lake format to obtain an ordered lake format file.

[0023] A further technical solution is that the Paimon interface is used to generate a snapshot ID reflecting the current database state based on the transaction start timestamp or the time field in the table, including:

[0024] Information is extracted from the start timestamp of the distributed transaction or the time field in the database table, and a unique snapshot ID is generated using the Paimon interface.

[0025] A further technical solution is that the standard data lake reader includes Flink, Spark and Doris.

[0026] The application also provides a distributed database incremental snapshot device, characterized by comprising:

[0027] An acquisition unit is configured to acquire a change log from a distributed database node, wherein the change log includes transaction key information.

[0028] An assembly unit is configured to perform transaction assembly on the change log to obtain an assembly result.

[0029] A sorting unit is configured to use the start timestamp of the transaction as a sorting key, and perform merge sorting on the assembly result through a log-structured merge tree structure to generate an ordered lake format file.

[0030] A snapshot unit is configured to use the Paimon interface to generate a snapshot ID reflecting the current database state based on the transaction start timestamp or the time field in the table.

[0031] A query unit is configured to query the ordered lake format file and the snapshot ID using a standard data lake reader.

[0032] The application also provides a computer device, characterized by comprising a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0033] Compared with the prior art, the present application has the beneficial effects that: the present application obtains the change log of the distributed database node and extracts the transaction key information therein, then assembles the change logs to obtain an ordered result, uses the transaction start timestamp as the sorting key and performs merge sorting on the assembled result through the LSM tree structure to generate an ordered lake format file, uses the Paimon interface to generate a snapshot ID reflecting the current database state according to the transaction start timestamp or the table time field based on the process, and finally queries the ordered lake format file and the snapshot ID by using the standard data lake reader, which not only simplifies the data migration process from the distributed database to the data lake, but also greatly improves the efficiency and reliability of the whole process. In addition, this method ensures the consistency and integrity of data transmission between different systems, because it relies on accurate timestamp sorting and efficient transaction management mechanism, which guarantees that even out-of-order change logs can be correctly sorted and applied to generate accurate snapshots, thereby supporting real-time data analysis and decision-making.

[0034] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0036] Figure 1 The application scenario diagram of the distributed database incremental snapshot method provided by the embodiment of the present application is shown in the figure.

[0037] Figure 2 The flowchart of the distributed database incremental snapshot method provided by the embodiment of the present application is shown in the figure.

[0038] Figure 3 The sub-flowchart of the distributed database incremental snapshot method provided by the embodiment of the present application is shown in the figure.

[0039] Figure 4 The schematic block diagram of the distributed database incremental snapshot device provided by the embodiment of the present application is shown in the figure.

[0040] Figure 5 The schematic block diagram of the sorting unit of the distributed database incremental snapshot device provided by the embodiment of the present application is shown in the figure.

[0041] Figure 6 The schematic block diagram of the computer device provided by the embodiment of the present application is shown in the figure.

[0042] Figure 7A schematic diagram of the LSMTree provided by the embodiments of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0044] It should be understood that, when used in the specification and the appended claims, the terms "comprise" and "include" indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms, unless the context clearly indicates otherwise.

[0046] It should be further understood that the term "and / or" used in the specification and the appended claims of the present application refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0047] Please refer to Figure 1 and Figure 2 , Figure 1 A schematic diagram of the application scenario of the distributed database incremental snapshot method provided by the embodiments of the present application. Figure 2The schematic flowchart of the distributed database incremental snapshot method provided by the embodiment of the present application. The distributed database incremental snapshot method is applied to a server. The server interacts with a terminal for data, and realizes simplification, efficiency improvement and reliability enhancement of the data migration process from the distributed database to the data lake through a series of carefully designed steps. First, it uses the RPC protocol to obtain the change log, and assembles and processes these logs according to the transaction key information, and then sorts these logs based on the transaction start time stamp using the LSM tree structure, to generate an ordered lake format file. Next, a unique snapshot ID is generated based on the transaction or the table time field using the Paimon interface to reflect the current database state. Finally, the ordered files and snapshot IDs are queried through a standard data lake reader such as Flink, Spark, and Doris. This method not only guarantees the consistency and integrity of the data during transmission between different systems, but also significantly improves the efficiency and reliability of the entire operation by optimizing the data processing process, making large-scale data migration more smooth and efficient.

[0048] Figure 2 The flowchart of the distributed database incremental snapshot method provided by the embodiment of the present application. As shown in Figure 2 the method comprises the following steps S110 to S150.

[0049] S110, obtaining a change log from a distributed database node, wherein the change log comprises transaction key information.

[0050] In this embodiment, the change log from the distributed database node is obtained through the RPC protocol.

[0051] The transaction key information includes unique identification, log type of prewrite, rollback or commit, operation type, start time stamp, commit time stamp, modified value and unmodified value.

[0052] Specifically, in order to realize data migration from the distributed database to the data lake, first, the change log from each distributed database node is obtained through the RPC (Remote Procedure Call) protocol. This process is one of the basic steps of the entire method, and its purpose is to collect all transaction-related change information, providing necessary data support for subsequent data processing and snapshot generation.

[0053] The change log contains the key information of the transaction, which is crucial for ensuring data consistency and integrity. Specifically, the change log includes the following aspects:

[0054] Unique identification (Key): used to uniquely identify each record or each operation, which is the basis for identifying different log records.

[0055] Log Type: Indicates whether the log record belongs to Prewrite, Rollback, or Committed. This helps to distinguish different transaction states and adopt corresponding processing strategies.

[0056] Op Type: Specifies the type of operation performed on the data, such as Put, Delete, etc. This is crucial for understanding how data is modified.

[0057] Start Ts: Marks the time point when a transaction starts. It is used not only for sorting but also for the uniqueness of the transaction.

[0058] Commit Ts: Records the time when the transaction is finally committed. This is crucial for confirming the completion of the transaction and its order.

[0059] Value: Represents the latest state of the data after the operation.

[0060] Old Value: Records the original state of the data before it was modified. This is very useful for recovery and auditing.

[0061] In practical applications, each distributed database node will be equipped with a data reader responsible for periodically polling and obtaining change logs on that node through the RPC protocol. These change logs will then be sent to the transaction assembler for further processing. It is worth noting that since a transaction may involve operations on multiple nodes, the collected logs may be out of order. However, by using the start timestamp as a key sorting field, this problem can be effectively solved in subsequent steps, ensuring data consistency and accuracy.

[0062] This method not only simplifies the process of migrating data from a distributed database to a data lake, but also improves the efficiency and reliability of the entire process by effectively utilizing key transaction information, while ensuring consistency and integrity of data during transmission between different systems.

[0063] S120, transaction assembly of the change log is performed to obtain an assembly result.

[0064] In this embodiment, the assembly result refers to the extraction and reorganization of all relevant change logs belonging to the same transaction from the change logs of the distributed database through specific processing steps, forming a complete transaction data list. Specifically, the assembly result contains the following key aspects:

[0065] For each change log, a unique intermediate identifier (Mid) is generated by combining its unique key (Key) and start timestamp (StartTs). This Mid is used to identify all change logs belonging to the same transaction.

[0066] Prewrite (pre-write) log: When a Prewrite type log is encountered, the change log is temporarily stored in the cache, waiting for a subsequent Committed or Rollback log to determine the final state of the transaction.

[0067] Committed log: Once a Committed type log is received, the system searches for all change logs associated with the current Mid in the cache and outputs these change logs as a complete transaction data list to the next stage (such as a transaction sorter) for further processing.

[0068] Rollback log: If a Rollback type log is received, it means that the corresponding transaction is canceled. At this time, any change logs associated with the current Mid in the cache will be deleted, as these changes will not take effect.

[0069] The assembly result ensures that all change logs of each transaction are correctly gathered together, even if these logs are initially out of order or come from different distributed database nodes. The purpose of this is to ensure the integrity and consistency of transactions and provide accurate data support for subsequent sorting, snapshot generation, and other operations.

[0070] After completing transaction assembly, the resulting transaction data list can be directly input into the transaction sorter, which uses the LSMTree structure to sort these data according to the start timestamp (StartTs) to generate an ordered data file in the Paimon data lake format.

[0071] In summary, in this embodiment, the assembly result is a carefully organized and reorganized transaction data list that contains all change logs belonging to the same transaction, and these logs have been processed according to their types to facilitate subsequent data processing and analysis work.

[0072] For each change log, extract the unique identifier and start timestamp to generate a Mid value, and classify and process according to the log type to obtain the assembly result.

[0073] When the log type is prewrite, the change log is written to a cache; when the log type is rollback, the change log corresponding to the Mid value is deleted from the cache; when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value to form a data list containing out-of-order change logs.

[0074] In this embodiment, in order to ensure that the change logs obtained from the distributed database nodes can correctly reflect the state of each transaction and provide accurate data support for subsequent data sorting and snapshot generation, it is necessary to perform transaction assembly on these change logs. The key of this step is to identify and reorganize all change logs belonging to the same transaction to facilitate subsequent processing.

[0075] For each change log, its unique identifier (Key) and start timestamp (StartTs) need to be extracted first. These two form a value called Mid, which is used to uniquely identify all related change logs in a transaction. For example, assuming that the Key of a change log is user_123 and its StartTs is 1625097600000 (i.e., 2021-07-01T00:00:00Z), then the Mid can be simply generated by concatenating these two fields, such as user_123_1625097600000.

[0076] Next, according to the log type (LogType) of the change log, they are classified and processed:

[0077] Prewrite (prewrite): If the log type is Prewrite, it indicates that it is part of a transaction, but the transaction has not been committed yet. At this time, this change log will be temporarily stored in a cache area, waiting for the corresponding Commit or Rollback log to arrive.

[0078] Committed (committed): Once a log of type Committed is received, it means that the transaction marked as Prewrite before has been successfully completed. At this time, the system will find all change logs related to the current Mid from the cache and output them as a complete transaction data list to the next stage (transaction sorter) for processing.

[0079] Rollback (rollback): If a log of type Rollback is received, it means that the corresponding transaction is canceled. In this case, any change log in the cache associated with the current Mid will be deleted, because these changes will not ultimately take effect.

[0080] After the above steps, the transaction assembly result composed of multiple change logs is obtained. This result is a list containing all change logs belonging to the same transaction. It should be noted that since these logs can come from different distributed database nodes, they can be out of order when entering the transaction assembler. However, during the transaction assembly process, all change logs belonging to the same transaction can be effectively gathered together by using Mid as the key, making preparations for the next sorting operation.

[0081] This method not only ensures the integrity of the transaction, but also provides the necessary input for subsequent efficient sorting using the LSM Tree structure. In addition, in this way, the problem of data inconsistency that may occur in a distributed environment can also be effectively addressed, thereby ensuring the reliability and consistency of the entire system.

[0082] S130, using the start timestamp of the transaction as the sorting key, performing merge sorting on the assembly result through the log-structured merge tree structure to generate an ordered lake format file.

[0083] In this embodiment, the ordered lake format file refers to a data file sorted according to a specific rule (in this case, the start timestamp of the transaction), which conforms to the Paimon data lake format specification and can be used for subsequent data analysis, snapshot generation, etc.

[0084] In an embodiment, please refer to Figure 3 The above step S130 can include steps S131-S133.

[0085] S131, set the start timestamp of the distributed transaction as the sorting key.

[0086] In this embodiment, first of all, the start timestamp (StartTs) of each transaction is used as the key field for sorting. This is done to ensure that all log entries belonging to the same transaction can be correctly arranged in chronological order, thereby ensuring the consistency and accuracy of transaction processing.

[0087] The reason for choosing the start timestamp as the sorting key is that it accurately reflects the time point when the transaction starts, which is crucial for the integrity and order of the transaction. In addition, in a distributed environment, ensuring that transactions are processed in the order of their actual occurrence is of great significance to maintaining global consistency.

[0088] S132, write the received out-of-order transaction change logs to layer 0 of the log-structured merge tree according to the Paimon data lake format, and explicitly specify the start timestamp as the sorting field.

[0089] In this embodiment, as Figure 7As shown, when receiving out-of-order transaction change logs from distributed database nodes, these logs are first written to the top level of the LSM Tree, Level 0. Here, each change log is converted into a data record conforming to the Paimon data lake format, with a start timestamp specifically designated as the sorting field.

[0090] During this process, all change logs need to be formatted according to the requirements of the Paimon data lake format. This means that in addition to containing the original change information, necessary metadata such as the start timestamp needs to be added to facilitate subsequent sorting and merging operations.

[0091] S133、When data is written to Level 0, a merge of the Log-Structured Merge Tree is triggered. During the merge process, the merge sort sorts the assembly results according to the start timestamp and reassembles the assembly results into files conforming to the data lake format to obtain ordered lake format files.

[0092] In this embodiment, once new data is written to Level 0 of the LSM Tree, a merge operation is triggered. In this step, the system compares data blocks in different levels and performs merge sorting based on the specified sorting field (here, the start timestamp).

[0093] Through the merge sorting algorithm, the system can effectively sort the originally unordered transaction change logs according to their start timestamps. After sorting, these log entries are reorganized into ordered files conforming to the Paimon data lake format.

[0094] Finally, the ordered files obtained through the above processing not only contain the correct transaction order, but also meet all the requirements of the Paimon data lake format, and can be directly used for data analysis, snapshot generation, or other related applications.

[0095] This LSM Tree-based data processing method not only improves data processing efficiency, but also ensures the consistency and order of transactions, which is of great significance for building real-time and large-scale database snapshots. At the same time, this method also fully utilizes the advantages of the Paimon data lake format, making the generated data files able to be efficiently read and analyzed in various query engines (such as Spark, Flink, etc.).

[0096] S140、Based on the transaction start timestamp or the table time field, use the Paimon interface to generate a snapshot ID reflecting the current database state.

[0097] In this embodiment, the snapshot ID is an identifier used to uniquely identify the database state at a specific moment. By using the start timestamp of a distributed transaction or the time field information in the database table, combined with the interface provided by the Paimon data lake format, a snapshot ID reflecting the current database state can be generated.

[0098] Specifically, information is extracted from the start timestamp of a distributed transaction or the time field in a database table, and a unique snapshot ID is generated using the Paimon interface.

[0099] First, the start timestamp (StartTs) is extracted from the log of each distributed transaction. This timestamp accurately records the time point when the transaction starts, which is crucial for ensuring the consistency and order of transactions.

[0100] If the start timestamp of a transaction cannot be directly obtained in some cases, or a more detailed state description is needed, specific time fields can be extracted from related database tables. These time fields are usually associated with specific business logic and provide more accurate time markers.

[0101] Once the time information for generating the snapshot ID is determined, the next step is to use the interface provided by Paimon to create a unique snapshot ID. Paimon, as a data lake format supporting real-time processing and batch processing, its API design aims to simplify data management and query processes, including snapshot management functions.

[0102] The snapshot ID is usually composed of time information and other unique identifiers. For example, it can be a string composed of the transaction start timestamp, database table name, and an incremental sequence number. This structure not only guarantees the uniqueness of the snapshot ID, but also facilitates subsequent management and query.

[0103] Each snapshot ID corresponds to the complete state of the database at a specific time point. This is of great significance for data analysis, backup and recovery, and historical data query scenarios. By specifying the snapshot ID, users can quickly locate and access the required historical data version.

[0104] By introducing the snapshot mechanism, not only can the storage space requirement be greatly reduced, but also the query efficiency can be significantly improved. Because when performing large-scale data analysis, directly operating the original data often takes a long time and is prone to errors; while using snapshots, accurate data views can be obtained in a more efficient way.

[0105] Whenever a new transaction is committed, the system will check whether it is necessary to generate a new snapshot ID based on the latest time information. If so, it will call the relevant interfaces of Paimon to perform the snapshot generation process.

[0106] To ensure the correctness and uniqueness of snapshot IDs, thorough testing is required before actual deployment. Additionally, considering the performance requirements of different application scenarios, the snapshot generation strategy may need to be adjusted and optimized accordingly.

[0107] In summary, the key to the S140 step lies in effectively utilizing the transaction start timestamp or time field information in the database table and generating reliable snapshot IDs with the powerful interface capabilities of Paimon, thereby achieving effective management and querying of database states. This process not only improves the flexibility and efficiency of data processing, but also provides a solid foundation for subsequent data analysis.

[0108] S150, using a standard data lake reader to query the ordered lake format file and the snapshot ID.

[0109] In this embodiment, the standard data lake reader includes Flink, Spark, and Doris.

[0110] The ordered lake format file and snapshot ID generated through the previous steps can be efficiently queried using standard data lake readers such as Flink, Spark, and Doris.

[0111] Flink: A stream processing framework that also supports batch processing, known for its low latency and high throughput. It seamlessly integrates Paimon data lake format, providing real-time data analysis capabilities.

[0112] Spark: A widely used distributed computing system that supports SQL queries, stream processing, and machine learning for advanced analysis functions. Spark is also tightly integrated with Paimon, allowing users to easily access and process large-scale data sets.

[0113] Doris: A modern MPP (Massively Parallel Processing) database designed for efficient OLAP (Online Analytical Processing). Doris' support for Paimon makes it an ideal choice for querying and analyzing Paimon data lake format.

[0114] First, the generated ordered lake format file needs to be loaded into the selected data lake reader. This usually involves specifying the correct path and file format (Paimon format in this case) so that the reader knows how to parse and process the data.

[0115] Once the data is correctly loaded, it can be queried using SQL statements or other programming interfaces. For example, in Spark SQL, you can write a query like SELECT * FROM lake_table WHERE snapshot_id='specific_snapshot_id' to retrieve data corresponding to a specific snapshot ID.

[0116] Each snapshot has a unique identifier (snapshot ID) that represents the state of the database at a certain point in time. To query the data state at a specific time point, the corresponding snapshot ID must be accurately identified.

[0117] With the snapshot ID, the snapshot can be directly accessed through the data lake reader. For example, in Flink, you may need to set up a job to point to a specific snapshot ID and start processing the data stream from there.

[0118] Given the need for large-scale data processing, some strategies should be adopted to optimize query performance. For example, pre-built indexes, reasonable partitioning, and parallel processing are effective methods to improve efficiency.

[0119] Since the data comes from a distributed environment, it is crucial to ensure fault tolerance during the query process. Modern data lake readers all provide built-in fault recovery mechanisms to ensure that the query task can be completed even in the case of network instability or hardware failure.

[0120] Choosing the right data lake reader not only improves the performance of the current query task, but also leaves enough room for future expansion. Whether it is to add new data sources or introduce more complex analysis models, it can be relatively easy to implement.

[0121] In summary, the S150 step aims to demonstrate how to effectively query the ordered lake format files and specific snapshot IDs generated by the previous steps using standard data lake readers such as Flink, Spark, and Doris. This step not only enables accurate access to historical data states, but also provides strong support for subsequent data analysis and business decision-making. In this way, enterprises can quickly extract valuable information from massive amounts of data and make more informed decisions.

[0122] The method of the embodiment obtains the change log of the distributed database through RPC request, assembles transactions, and uses the Paimon data lake LSM Tree data structure to sort and generate Paimon data lake format data to restore the database operation and generate snapshot data. Finally, the snapshot data is queried through a query engine such as Spark, Flink, etc., to realize real-time and large-scale database snapshot. A distributed database is a database system that stores data on multiple physical nodes, which are usually connected together through a network. Unlike traditional centralized databases, distributed databases provide higher scalability, availability, and performance by distributing data across multiple nodes. Transactions are a method for database to guarantee data consistency, which are generally implemented based on the Percolator algorithm. The Paimon data lake format is a data lake format that supports the construction of real-time lake warehouse architecture using Flink and Spark, which can handle streaming and batch processing operations at the same time, and innovatively combines data lake format with LSM (Log-Structured Merge) structure, introducing real-time streaming updates into the lake warehouse architecture.

[0123] The method of the embodiment directly converts the change log of the distributed database into the data lake data format, and uses the snapshot method of the data lake to snapshot the data. Specifically, by reading the change log of the distributed database, assembling transactions, and using the LSM Tree lake format for sorting, data generation, and snapshot generation. The produced lake format file can be directly used by the lake format reader for data analysis, data backup, data cleaning, etc. First, the data reader reads the change log of the distributed database through the RPC protocol, and one distributed database node corresponds to one reader. The change log is composed of multiple fields such as Key, Logtype, Optype, startTs, CommitTs, Value, OldValue, etc., wherein Key is the unique key of the database log; Logtype includes prewrite, rollback, and committed to form the log type; Optype is composed of Put and Delete to form the operation type; startTs is the start time, commitTs is the commit time, value is the modified value, and OldValue is the value before modification. Then, the transaction assembler receives data from multiple distributed database nodes, and assembles the data of the same transaction into a data list. Since these data may come from different readers, the data in the list may be out of order. By extracting Key+StartTs to generate Mid, and according to Logtype to decide whether to write to cache or match all change logs of the current Mid from the cache to complete transaction assembly and write to the sorter.

[0124] The transaction sequencer receives the out-of-order transactions with the distributed transaction StartTs as the sorting key. After the out-of-order transactions are written into the transaction sequencer, the transaction sequencer writes a transaction change log in the Paimon data lake format to the Level 0 layer of the LSMTree and specifies the sorting field as StartTs. This triggers a merge of the LSM Tree, and the out-of-order data is sorted during the LSM Tree merge process using the Paimon data lake format merge sort, and then assembled into a lake format file and written to the storage. The snapshot generator extracts the snapshot ID from the distributed transaction StartTs or the time field in the distributed database table to generate a snapshot. The snapshot generation method uses the snapshot interface of Paimon itself. If the out-of-order data is not sorted correctly, for example, the operation order on the database is Delete, Insert, but after being out of order, it becomes Insert, Delete, then the data read by the reader is not accurate. Finally, the lake format file generated by the transaction sequencer and the snapshot file generated by the snapshot generator can be directly provided to a standard data lake reader (such as Flink, Spark, Doris) for reading according to the Paimon data format, and the data of the distributed database can be read therefrom, or provided to a downstream system for data reading or analysis. Open source tools such as Flink, Spark, Doris, etc. can be used to read these data.

[0125] The embodiment proposes a complete method of generating Paimon data lake format files and snapshot files using distributed database change logs, sorting the out-of-order distributed transactions using the Paimon data lake format LSM Tree, extracting the time field in the transaction change log of the change log, generating Paimon snapshot data, and providing it to the downstream lake format reader; combining the out-of-order data of the distributed database change log, using the data merge sorting capability of the Paimon data lake format LSM Tree, sorting the data during the generation of the Paimon data lake LSM Tree merge process, and triggering the submission and Compact of the LSM Tree once for each transaction submission, so as to ensure that the downstream can be accessed after the data processing is completed, and the real-time performance of the database snapshot is guaranteed; the transaction StartTs or the time field of the distributed database change log is used to generate the Paimon data lake format snapshot logic, and the snapshot of the incremental data is innovatively generated, which greatly saves the storage space.

[0126] The method utilizes the snapshot technology of the data lake to take snapshots of the data. Specifically, the present scheme involves reading the change logs of the distributed database, assembling transactions, and sorting these change logs using the LSM Tree structure to generate ordered data lake format files, so that the generated lake format files can be directly used by standard data lake readers, suitable for scenarios such as data analysis, data backup, and data cleaning. In addition, the method of the present embodiment emphasizes the ability to sort the possible out-of-order of distributed transactions using the LSM Tree feature of the Paimon data lake format. Further, the method of the present embodiment also proposes a method of implementing data sorting in the LSM Tree merging process in generating the Paimon data lake in combination with the out-of-order problem that may exist in the distributed database change logs, and the LSM Tree commit and Compact operation is triggered once per transaction commit, ensuring that the data can be used by downstream systems immediately after processing is completed, guaranteeing the real-time nature of the database snapshot. Finally, the method of the present embodiment proposes to use the transaction start timestamp (StartTs) in the distributed database change logs or the time field within the table to generate the snapshot logic of the Paimon data lake format, which not only greatly saves storage space, but also provides a new idea for incremental data snapshots, providing convenience for subsequent data processing. The entire process ensures that the data migration from the distributed database to the data lake is both simplified and efficient, while guaranteeing the consistency and integrity of the data during transmission between different systems.

[0127] The above-mentioned distributed database incremental snapshot method obtains the change logs of the distributed database nodes and extracts the transaction key information therein, then assembles the change logs by transactions to obtain ordered results, uses the transaction start timestamp as the sorting key and performs merge sorting on the assembled results through the LSM tree structure to generate ordered lake format files, generates a snapshot ID reflecting the current database state based on the transaction start timestamp or the table time field using the Paimon interface, and finally queries the ordered lake format files and snapshot ID using a standard data lake reader, which not only simplifies the data migration process from the distributed database to the data lake, but also greatly improves the efficiency and reliability of the entire process. In addition, this method ensures the consistency and integrity of data transmission between different systems, as it relies on accurate timestamp sorting and efficient transaction management mechanisms to ensure that even out-of-order change logs can be correctly sorted and applied to generate accurate snapshots, thereby supporting real-time data analysis and decision-making.

[0128] Figure 4 is a schematic block diagram of a distributed database incremental snapshot device 300 provided by an embodiment of the present application. As shown in Figure 4As shown, corresponding to the above-described distributed database incremental snapshot method, the present invention also provides a distributed database incremental snapshot apparatus 300. This distributed database incremental snapshot apparatus 300 includes a unit for executing the above-described distributed database incremental snapshot method, and the apparatus can be configured in a server. Specifically, please refer to... Figure 4 The distributed database incremental snapshot device 300 includes an acquisition unit 301, an assembly unit 302, a sorting unit 303, a snapshot unit 304, and a query unit 305.

[0129] The acquisition unit 301 is used to acquire change logs from distributed database nodes, wherein the change logs include key transaction information; the assembly unit 302 is used to assemble the change logs into transactions to obtain an assembly result; the sorting unit 303 is used to use the start timestamp of the transaction as the sorting key and to merge and sort the assembly result through a log structure merge tree structure to generate an ordered lake format file; the snapshot unit 304 is used to generate a snapshot ID reflecting the current database state using the Paimon interface based on the transaction start timestamp or the table time field; and the query unit 305 is used to query the ordered lake format file and the snapshot ID using a standard data lake reader.

[0130] In one embodiment, the acquisition unit 301 is used to acquire change logs from distributed database nodes via the RPC protocol.

[0131] In one embodiment, the assembly unit 302 is used to extract a unique identifier and a start timestamp for each change log to generate a Mid value, and to classify the logs according to their types to obtain an assembly result.

[0132] In one embodiment, the assembly unit 302 is further configured to: write the change log to the cache when the log type is prewrite; delete the change log with the corresponding Mid value from the cache when the log type is rollback; and match and assemble all change logs belonging to the same transaction from the cache according to the Mid value when the log type is committed, forming a data list containing out-of-order change logs.

[0133] In one embodiment, such as Figure 5 As shown, the sorting unit 303 includes a setting subunit 3031, a writing subunit 3032, and a sorting and merging subunit 3033.

[0134] The setting sub-unit 3031 is configured to set the start timestamp of the distributed transaction as a sorting key; the writing sub-unit 3032 is configured to write the received disordered transaction change log into the 0 layer of the log-structured merge tree in the Paimon data lake format, and explicitly specify the start timestamp as a sorting field; and the sorting and merging sub-unit 3033 is configured to trigger a merge of the log-structured merge tree when data is written into the 0 layer, and in the merging process, the sorting and merging sub-unit 3033 sorts the assembly result according to the start timestamp, and reassembles the assembly result into a file in the data lake format to obtain an ordered lake format file.

[0135] In an embodiment, the snapshot unit 304 is configured to extract information from the start timestamp of the distributed transaction or a time field in a database table, and generate a unique snapshot ID using a Paimon interface.

[0136] It should be noted that the specific implementation process of the distributed database incremental snapshot device 300 and each unit can be clearly understood by those skilled in the art, and can refer to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, details are not repeated here.

[0137] The distributed database incremental snapshot device 300 can be implemented in the form of a computer program, which can run on a computer device as shown in the computer device. Figure 6

[0138] Please refer to Figure 6 , Figure 6 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 can be a server, wherein the server can be a stand-alone server or a server cluster composed of multiple servers.

[0139] Referring to Figure 6 , the computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501, wherein the memory can include a non-volatile storage medium 503 and an internal memory 504.

[0140] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which when executed, can cause the processor 502 to perform a distributed database incremental snapshot method.

[0141] The processor 502 is configured to provide computing and control capabilities to support the operation of the entire computer device 500.

[0142] ​The memory 504 provides an environment for running the computer program 5032 in the non-volatile storage medium 503, which, when executed by the processor 502, can cause the processor 502 to perform a distributed database incremental snapshot method.

[0143] The network interface 505 is used for network communication with other devices. Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 500 to which the scheme of the present application is applied. The specific computer device 500 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] The processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps:

[0145] Obtain a change log from a distributed database node, wherein the change log includes transaction key information; transactionally assemble the change log to obtain an assembly result; use a start timestamp of a transaction as a sorting key to perform merge sorting on the assembly result through a log-structured merge tree structure to generate an ordered lake format file; use a Paimon interface based on a transaction start timestamp or an intra-table time field to generate a snapshot ID reflecting a current database state; and use a standard data lake reader to query the ordered lake format file and the snapshot ID.

[0146] The transaction key information includes a unique identifier, a log type of prewrite, rollback or commit, an operation type, a start timestamp, a commit timestamp, a modified value and a modified value.

[0147] The standard data lake reader includes Flink, Spark and Doris.

[0148] In an embodiment, the processor 502, when implementing the step of obtaining a change log from a distributed database node, specifically implements the following steps:

[0149] Obtain a change log from a distributed database node through an RPC protocol.

[0150] In an embodiment, the processor 502, when implementing the step of transactionally assembling the change log to obtain an assembly result, specifically implements the following steps:

[0151] For each change log, extract a unique identifier and a start timestamp to generate a Mid value, and perform classification processing according to a log type to obtain an assembly result.

[0152] In an embodiment, the processor 502 implements the following steps when implementing the step of classifying according to the log type:

[0153] When the log type is prewrite, the change log is written into the cache; when the log type is rollback, the change log corresponding to the Mid value is deleted from the cache; when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value to form a data list containing out-of-order change logs.

[0154] In an embodiment, the processor 502 implements the following steps when implementing the step of merging the assembly result by using the start timestamp of the transaction as the sorting key through the log-structured merge tree structure to generate an ordered lake format file:

[0155] The start timestamp of the distributed transaction is set as the sorting key; the received out-of-order transaction change log is written into the 0 layer of the log-structured merge tree according to the Paimon data lake format, and the start timestamp is explicitly specified as the sorting field; when the data is written into the 0 layer, a merge of the log-structured merge tree is triggered once, and in the merging process, the assembly result is sorted according to the start timestamp, and the assembly result is reassembled into a file conforming to the data lake format to obtain an ordered lake format file.

[0156] In an embodiment, the processor 502 implements the following steps when implementing the step of generating a snapshot ID reflecting the current database state based on the start timestamp of the transaction or the time field in the table using the Paimon interface:

[0157] Information is extracted from the start timestamp of the distributed transaction or the time field in the database table, and a unique snapshot ID is generated using the Paimon interface.

[0158] It should be understood that in the embodiments of the present application, the processor 502 can be a central processing unit (CPU), and the processor 502 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0159] Those skilled in the art can understand that all or part of the processes in the method of implementing the above embodiments can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments of the method.

[0160] Therefore, the application also provides a storage medium. The storage medium can be a computer readable storage medium. The storage medium stores a computer program, wherein the computer program is executed by a processor to make the processor execute the following steps:

[0161] obtaining a change log from a distributed database node, wherein the change log includes transaction key information; transaction assembling the change log to obtain an assembly result; using the start timestamp of the transaction as a sorting key, performing merge sorting on the assembly result through a log structure merge tree structure to generate an ordered lake format file; using a Paimon interface based on the transaction start timestamp or an intra-table time field to generate a snapshot ID reflecting the current database state; using a standard data lake reader to query the ordered lake format file and the snapshot ID.

[0162] The transaction key information includes a unique identifier, a pre-write, a rollback or a committed log type, an operation type, a start timestamp, a commit timestamp, a modified value and a modified value.

[0163] The standard data lake reader includes Flink, Spark and Doris.

[0164] In an embodiment, the processor, when executing the computer program to implement the step of obtaining the change log from the distributed database node, specifically implements the following steps:

[0165] The change log from the distributed database node is obtained through an RPC protocol.

[0166] In an embodiment, the processor, when executing the computer program to implement the step of transaction assembling the change log to obtain an assembly result, specifically implements the following steps:

[0167] For each change log, the unique identifier and the start timestamp are extracted to generate a Mid value, and the log type is classified to obtain an assembly result.

[0168] In an embodiment, the processor, when executing the computer program to implement the step of classifying processing according to the log type, specifically implements the following steps:

[0169] When the log type is prewrite, the change log is written into the cache; when the log type is rollback, the change log corresponding to the Mid value is deleted from the cache; and when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value to form a data list containing out-of-order change logs.

[0170] In an embodiment, when the processor executes the computer program to implement the step of using the start timestamp of a transaction as a sorting key, performing merge sort on the assembly result through a log-structured merge tree structure to generate an ordered lake format file, the following steps are implemented:

[0171] The start timestamp of a distributed transaction is set as a sorting key; the received out-of-order transaction change log is written into the 0 layer of a log-structured merge tree in the Paimon data lake format, and the start timestamp is explicitly specified as a sorting field; when data is written into the 0 layer, a merge of the log-structured merge tree is triggered, and in the merge process, the merge sort sorts the assembly result according to the start timestamp, and reassembles the assembly result into a file conforming to the data lake format to obtain an ordered lake format file.

[0172] In an embodiment, when the processor executes the computer program to implement the step of generating a snapshot ID reflecting the current database state based on the transaction start timestamp or the table internal time field using the Paimon interface, the following steps are implemented:

[0173] Generating a snapshot ID reflecting the current database state based on the transaction start timestamp or the table internal time field using the Paimon interface

[0174] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer readable storage media that can store program codes.

[0175] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0176] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the embodiments of the apparatus described above are merely schematic. For example, the division of the units is merely a logical function division. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In this way, the inventive idea can be implemented.

[0177] The steps in the method embodiments of the present application can be adjusted, combined and deleted in sequence according to actual needs. The units in the apparatus embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0178] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application.

[0179] The above describes only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for distributed database incremental snapshotting, characterized in that, The method comprises the following steps: obtaining change logs from distributed database nodes, wherein the change logs comprise transaction key information; transactionally assembling the change logs to obtain an assembly result; performing merge sorting on the assembly result by using a log-structured merge tree structure and taking the start timestamp of a transaction as a sorting key to generate an ordered lake format file; generating a snapshot ID reflecting the current database state based on the start timestamp of the transaction or a time field in a table using a Paimon interface; querying the ordered lake format file and the snapshot ID by using a standard data lake reader; the transactionally assembling the change logs to obtain an assembly result comprises: for each of the change logs, extracting a unique identifier and a start timestamp to generate a Mid value, and performing classification processing according to the log type to obtain an assembly result; the classification processing according to the log type comprises: when the log type is prewrite, the change log is written into a cache; when the log type is rollback, the change log corresponding to the Mid value is deleted from the cache; and when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value to form a data list containing disordered change logs; the performing merge sorting on the assembly result by using a log-structured merge tree structure and taking the start timestamp of a transaction as a sorting key to generate an ordered lake format file comprises: setting the start timestamp of a distributed transaction as a sorting key; writing the received disordered transaction change logs into layer 0 of the log-structured merge tree according to the Paimon data lake format, and explicitly specifying the start timestamp as a sorting field; when the data is written into layer 0, triggering a merge of the log-structured merge tree, and in the merging process, the merge sorting sorts the assembly result according to the start timestamp, and reassembles the assembly result into a file conforming to the data lake format to obtain an ordered lake format file.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining change logs from distributed database nodes by using an RPC protocol.

3. The method of claim 1, wherein, The transaction key information comprises a unique identifier, a log type of prewrite, rollback or committed, an operation type, a start timestamp, a commit timestamp, a modified value and a value before modification.

4. The method of claim 1, wherein, The method comprises the following steps: extracting information from the start timestamp of a distributed transaction or a time field in a database table, and generating a unique snapshot ID using a Paimon interface.

5. The method of claim 1, wherein, The standard data lake reader comprises Flink, Spark and Doris.

6. A distributed database incremental snapshot apparatus, characterized by, The device uses the distributed database incremental snapshot method according to any one of claims 1 to 5, which comprises: an obtaining unit configured to obtain change logs from distributed database nodes, wherein the change logs comprise transaction key information; an assembling unit configured to transactionally assemble the change logs to obtain an assembly result; An ordering unit is configured to use the transaction start timestamp as a sorting key, and perform merge sort on the assembly result through a log-structured merge tree structure to generate an ordered lake format file; A snapshot unit is configured to use a Paimon interface to generate a snapshot ID reflecting a current database state based on a transaction start timestamp or an in-table time field; A query unit is configured to query the ordered lake format file and the snapshot ID using a standard data lake reader.

7. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 5 when executing the computer program.

Images