Distributed database incremental snapshot method and device and computer equipment

Abstract

The invention discloses a distributed database incremental snapshot method and device and computer equipment. The method comprises the steps that a change log from a distributed database node is obtained, and the change log comprises transaction key information; performing transaction assembly on the change log to obtain an assembly result; using a starting timestamp of the transaction as a sorting key, and merging and sorting the assembly results through a log structure merging tree structure to generate an ordered lake format file; generating a snapshot ID reflecting a current database state by using a Paimon interface based on a transaction starting timestamp or a time field in a table; and querying the ordered lake format file and the snapshot ID by using a standard data lake reader. By implementing the method provided by the invention, the data migration process from the distributed database to the data lake can be simplified, the efficiency and reliability of the whole process are improved, and the consistency and integrity of the data during transmission among different systems can be ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to a database processing method, in particular to a distributed database incremental snapshot method, device and computer equipment. BACKGROUND

[0002] A distributed database is an innovative database system that provides higher scalability, availability and performance by distributing data across multiple physical nodes connected by a network. Compared with traditional centralized databases, this approach not only improves data processing efficiency, but also enhances the fault tolerance of the system. In a distributed database, transactions are the core mechanism to ensure data consistency, usually based on the Percolator algorithm, which can effectively support transaction management in a large-scale data environment. On the other hand, the paimon data lake format, as an emerging data storage solution, combines the advantages of streaming and batch processing operations, and combines data lake format with LSM (Log-Structured Merge) structure, realizing support for real-time data updates.

[0003] Currently, in application scenarios requiring high accuracy and high performance, distributed databases are widely used, especially in cases where data needs to be exported to a data lake for backup or cleaning. However, existing methods either rely on SQL to read data and write to a data lake, or import data into a data lake by reading the change log of a distributed database, which has limitations in terms of operation complexity and efficiency; and the existing technology realizes cross-node consistent snapshot through global transaction management and recording transaction state, which not only increases the complexity and resource consumption of the system, but also may cause performance bottlenecks and scalability problems, especially when handling high concurrency and large-scale data; at the same time, the data import scheme relying on a specific change log format limits flexibility and the recovery process is complex in the event of a failure, these factors together constitute the main challenges in implementing efficient, flexible and high-performance distributed database applications. Therefore, it is necessary to design a new method to not only simplify the data migration process from a distributed database to a data lake, but also improve the efficiency and reliability of the entire process, and ensure the consistency and integrity of data transmission between different systems. SUMMARY

[0004] The purpose of the present application is to overcome the defects of the prior art and provide a distributed database incremental snapshot method, device and computer equipment.

[0005] To achieve the above purpose, the present application adopts the following technical solution: a distributed database incremental snapshot method, comprising: Obtaining a change log from a distributed database node, wherein the change log comprises transaction key information; Transaction assembling the change log to obtain an assembly result; Using the start timestamp of the transaction as the sorting key, the assembly result is sorted by the log structure merge tree structure to generate an ordered lake format file; Based on the transaction start timestamp or the table time field, use the Paimon interface to generate a snapshot ID reflecting the current database state; Using a standard data lake reader to query the ordered lake format file and the snapshot ID.

[0006] Further technical solutions are as follows: the change log obtained from the distributed database node comprises: Obtaining a change log from a distributed database node through an RPC protocol.

[0007] Further technical solutions are as follows: the transaction key information comprises a unique identifier, a prewrite, a rollback or a committed log type, an operation type, a start timestamp, a commit timestamp, a modified value and a modified value.

[0008] Further technical solutions are as follows: the transaction assembling the change log to obtain an assembly result comprises: For each change log, extract the unique identifier and the start timestamp to generate the Mid value, and perform classification processing according to the log type to obtain the assembly result.

[0009] Further technical solutions are as follows: the classification processing according to the log type comprises: When the log type is prewrite, the change log is written into the cache; when the log type is rollback, the change log corresponding to the Mid value is deleted from the cache; when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value, and a data list containing disordered change logs is formed.

[0010] Further technical solutions are as follows: the use of the start timestamp of the transaction as the sorting key, the sorting of the assembly result by the log structure merge tree structure to generate an ordered lake format file comprises: Set the start timestamp of the distributed transaction as the sorting key; Write the received disordered transaction change log into the 0 layer of the log structure merge tree according to the Paimon data lake format, and explicitly specify the start timestamp as the sorting field; When data is written to the 0 layer, a merge of the log-structured merge tree is triggered, and during the merge process, the assembly result is sorted according to the start timestamp by merge sort, and the assembly result is reassembled into a file conforming to the data lake format to obtain an ordered lake format file.

[0011] Further technical solutions are as follows: the Paimon interface is used to generate a snapshot ID reflecting the current database state based on a transaction start timestamp or a table time field, and the snapshot ID includes: Information is extracted from a transaction start timestamp or a time field in a database table, and a unique snapshot ID is generated using the Paimon interface.

[0012] Further technical solutions are as follows: the standard data lake reader includes Flink, Spark, and Doris.

[0013] The application further provides a distributed database incremental snapshot device, characterized by comprising: An acquisition unit is configured to acquire a change log from a distributed database node, wherein the change log includes transaction key information. An assembly unit is configured to perform transaction assembly on the change log to obtain an assembly result. A sorting unit is configured to use a transaction start timestamp as a sorting key, perform merge sorting on the assembly result through a log-structured merge tree structure, and generate an ordered lake format file. A snapshot unit is configured to use the Paimon interface to generate a snapshot ID reflecting the current database state based on a transaction start timestamp or a table time field. A query unit is configured to use a standard data lake reader to query the ordered lake format file and the snapshot ID.

[0014] The application further provides a computer device, characterized by comprising a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0015] Compared with the prior art, the present application has the beneficial effects that: the present application obtains the change log of the distributed database node and extracts the transaction key information therein, then assembles the change logs to obtain an ordered result, uses the transaction start timestamp as the sorting key and performs merge sorting on the assembled result through the LSM tree structure to generate an ordered lake format file, uses the Paimon interface to generate a snapshot ID reflecting the current database state according to the transaction start timestamp or the table time field based on the process, and finally uses the standard data lake reader to query the ordered lake format file and the snapshot ID, which not only simplifies the data migration process from the distributed database to the data lake, but also greatly improves the efficiency and reliability of the whole process. In addition, this method ensures the consistency and integrity of data transmission between different systems, because it relies on accurate timestamp sorting and efficient transaction management mechanism, ensuring that even out-of-order change logs can be correctly sorted and applied to generate accurate snapshots, thereby supporting real-time data analysis and decision-making.

[0016] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The application scenario of the distributed database incremental snapshot method provided by the embodiment of the present application is shown in the figure. Figure 2 The flowchart of the distributed database incremental snapshot method provided by the embodiment of the present application is shown in the figure. Figure 3 The sub-flowchart of the distributed database incremental snapshot method provided by the embodiment of the present application is shown in the figure. Figure 4 The schematic block diagram of the distributed database incremental snapshot device provided by the embodiment of the present application is shown in the figure. Figure 5 The schematic block diagram of the sorting unit of the distributed database incremental snapshot device provided by the embodiment of the present application is shown in the figure. Figure 6 The schematic block diagram of the computer device provided by the embodiment of the present application is shown in the figure. Figure 7 The schematic diagram of the LSMTree provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0020] It should be understood that the terms "comprising" and "including" as used in the specification and the appended claims indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0021] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, "a", "an", and "the" in singular form are intended to include plural forms unless the context clearly indicates otherwise.

[0022] It should be further understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0023] Please refer to Figure 1 and Figure 2 , Figure 1 The application scenario diagram of the distributed database incremental snapshot method provided by the embodiments of the present application. Figure 2 The schematic flowchart of the distributed database incremental snapshot method provided by the embodiments of the present application. The distributed database incremental snapshot method is applied to a server. The server interacts with a terminal for data, and realizes simplification, efficiency improvement and reliability enhancement of the data migration process from a distributed database to a data lake through a series of carefully designed steps. First, it uses the RPC protocol to obtain change logs, and assembles and processes these logs according to transaction key information, and then sorts these logs based on transaction start time stamps using the LSM tree structure to generate ordered lake format files. Next, a unique snapshot ID is generated based on the transaction or the table time field using the Paimon interface to reflect the current database state. Finally, the ordered files and snapshot IDs are queried through a standard data lake reader such as Flink, Spark, and Doris. This method not only guarantees the consistency and integrity of data transmission between different systems, but also significantly improves the efficiency and reliability of the entire operation by optimizing the data processing process, making large-scale data migration more smooth and efficient.

[0024] Figure 2 is a flowchart of a distributed database incremental snapshot method provided by an embodiment of the present application. As shown in Figure 2 , the method comprises the following steps S110-S150.

[0025] S110, obtaining a change log from a distributed database node, wherein the change log comprises transaction key information.

[0026] In this embodiment, the change log from the distributed database node is obtained through the RPC protocol.

[0027] The transaction key information includes unique identification, prewrite, rollback or committed log type, operation type, start timestamp, commit timestamp, modified value and unmodified value.

[0028] Specifically, in order to realize data migration from a distributed database to a data lake, first of all, the change log from each distributed database node needs to be obtained through the RPC (Remote Procedure Call) protocol. This process is one of the basic steps of the entire method, and its purpose is to collect all transaction-related change information, providing necessary data support for subsequent data processing and snapshot generation.

[0029] The change log contains the key information of the transaction, which is crucial for ensuring data consistency and integrity. Specifically, the change log includes the following aspects: Unique identification (Key): used to uniquely identify each record or each operation, which is the basis for identifying different log records.

[0030] Log type (Log Type): indicates whether the log record belongs to prewrite (Prewrite), rollback (Rollback) or committed (Committed). This helps to distinguish different transaction states and take appropriate processing strategies.

[0031] Operation type (Op Type): indicates the type of operation performed on the data, such as insert (Put), delete (Delete), etc. This is very important for understanding how data is modified.

[0032] Start timestamp (Start Ts): marks the time point when a transaction starts, which is not only used for sorting, but also for the uniqueness of transaction identification.

[0033] Commit timestamp (Commit Ts): records the time when the transaction is finally committed, which is crucial for confirming the completion of the transaction and its order.

[0034] Modified Value: Represents the latest state of data after operations.

[0035] Old Value: Records the original state of data before modification, which is useful for recovery and auditing.

[0036] In practical applications, each distributed database node is equipped with a data reader responsible for regularly polling and obtaining change logs on that node through RPC protocol. These change logs are then sent to the transaction assembler for further processing. It is worth noting that since transactions may involve operations on multiple nodes, the collected logs may be out of order. However, by using the start timestamp as a key sorting field, this problem can be effectively solved in subsequent steps, ensuring data consistency and accuracy.

[0037] This method not only simplifies the process of migrating data from distributed databases to data lakes, but also improves the efficiency and reliability of the entire process by effectively utilizing transaction key information, while ensuring data consistency and integrity during transmission between different systems.

[0038] S120, transaction assembly of the change log to obtain an assembly result.

[0039] In this embodiment, the assembly result refers to extracting and reorganizing all relevant change logs belonging to the same transaction from the change logs of the distributed database through specific processing steps to form a complete transaction data list. Specifically, the assembly result includes the following key aspects: For each change log, a unique intermediate identifier (Mid) is generated by combining its unique key (Key) and start timestamp (StartTs). This Mid is used to identify all change logs belonging to the same transaction.

[0040] Prewrite (Prewrite) log: When encountering a Prewrite type log, this change log is temporarily stored in the cache, waiting for the subsequent Committed or Rollback log to determine the final state of the transaction.

[0041] Committed (Committed) log: Once a Committed type log is received, the system will find all change logs related to the current Mid in the cache and output these change logs as a complete transaction data list to the next stage (such as the transaction sorter) for further processing.

[0042] Rollback log: If a log of type Rollback is received, it means that the corresponding transaction was undone. At this point, any change log in the cache associated with the current Mid will be deleted, as these changes will not take effect.

[0043] The assembly result ensures that all change logs of each transaction are correctly gathered together, even if these logs are initially out of order or come from different distributed database nodes. The purpose of this is to guarantee the integrity and consistency of transactions and provide accurate data support for subsequent sorting, snapshot generation, etc.

[0044] After completing transaction assembly, the resulting transaction data list can be directly input into the transaction sorter, which uses the LSMTree structure to sort these data according to the start timestamp (StartTs) to generate an ordered data file in the Paimon data lake format.

[0045] In summary, in this embodiment, the assembly result is a carefully organized and reorganized transaction data list that contains all change logs belonging to the same transaction, and these logs have been processed according to their types to facilitate subsequent data processing and analysis.

[0046] For each change log, the unique identifier and start timestamp are extracted to generate the Mid value, and the log is classified and processed according to the log type to obtain the assembly result.

[0047] When the log type is prewrite, the change log is written to the cache; when the log type is rollback, the change log corresponding to the Mid value is deleted from the cache; when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value to form a data list containing out-of-order change logs.

[0048] In this embodiment, in order to ensure that the change logs obtained from the distributed database nodes correctly reflect the state of each transaction and provide accurate data support for subsequent data sorting and snapshot generation, transaction assembly is required for these change logs. The key to this step is to identify and reorganize all change logs belonging to the same transaction to facilitate subsequent processing.

[0049] For each change log, the unique key (Key) and the start timestamp (StartTs) need to be extracted first. These two combined form a value called Mid, which is used to uniquely identify all related change logs in a transaction. For example, if a change log has a Key of user_123 and a StartTs of 1625097600000 (i.e., 2021-07-01T00:00:00Z), the Mid can simply be generated by concatenating these two fields, like user_123_1625097600000.

[0050] Next, the change logs are classified according to their log type (LogType): Prewrite: If the log type is Prewrite, it means this is part of a transaction, but the transaction has not yet been committed. At this time, this change log is temporarily stored in a cache area, waiting for the corresponding Commit or Rollback log to arrive.

[0051] Committed: Once a Committed type log is received, it means that the previously marked Prewrite transaction has been successfully completed. At this time, the system will find all change logs related to the current Mid from the cache and output them as a complete transaction data list to the next stage (transaction sorter) for processing.

[0052] Rollback: If a Rollback type log is received, it means that the corresponding transaction was canceled. In this case, any change logs associated with the current Mid in the cache will be deleted, as these changes will ultimately not take effect.

[0053] After the above steps, the transaction assembly result composed of multiple change logs is obtained. This result is a list containing all change logs belonging to the same transaction. It should be noted that since these logs may come from different distributed database nodes, they may be out of order when entering the transaction assembler. However, during the transaction assembly process, by using Mid as the key, all change logs belonging to the same transaction can be effectively aggregated together, preparing for the next sorting operation.

[0054] This method not only ensures the integrity of the transaction, but also provides the necessary input for efficient sorting using the LSM Tree structure in the next step. In addition, in this way, the problem of data inconsistency that may occur in a distributed environment can also be effectively addressed, thereby ensuring the reliability and consistency of the entire system.

[0055] S130, merge sort the assembly result by using the start timestamp of the transaction as the sorting key through the log-structured merge tree structure to generate an ordered lake format file.

[0056] In this embodiment, the ordered lake format file refers to a data file sorted according to a specific rule (in this case, the start timestamp of the transaction), which conforms to the Paimon data lake format specification and can be used for subsequent data analysis, snapshot generation, and other operations.

[0057] In an embodiment, referring to Figure 3 The step S130 described above can include steps S131-S133.

[0058] S131, set the start timestamp of the distributed transaction as the sorting key.

[0059] In this embodiment, first, the start timestamp (StartTs) of each transaction is used as the key field for sorting. This is done to ensure that all log entries belonging to the same transaction can be correctly arranged in chronological order, thereby ensuring the consistency and accuracy of transaction processing.

[0060] The reason for choosing the start timestamp as the sorting key is that it accurately reflects the time point when the transaction starts, which is crucial for the integrity and order of the transaction. In addition, in a distributed environment, ensuring that transactions are processed in the order of their actual occurrence is of great significance to maintaining global consistency.

[0061] S132, write the received out-of-order transaction change log to level 0 of the log-structured merge tree according to the Paimon data lake format, and specify the start timestamp as the sorting field.

[0062] In this embodiment, as shown in Figure 7 When the out-of-order transaction change logs are received from the distributed database nodes, these logs are first written to the uppermost layer of the LSM Tree, i.e., Level 0. Here, each change log is converted into a data record conforming to the Paimon data lake format, and the start timestamp is specifically specified as the sorting field.

[0063] In this process, all change logs need to be formatted according to the requirements of the Paimon data lake format. This means that in addition to containing the original change information, necessary metadata such as the start timestamp needs to be added to facilitate subsequent sorting and merging operations.

[0064] S133、When data is written to Level 0, a merge of the log-structured merge tree is triggered. During the merge process, the merge sort sorts the assembly results according to the start timestamp and reassembles the assembly results into a file conforming to the data lake format to obtain an ordered lake format file.

[0065] In this embodiment, once new data is written to Level 0 of the LSM Tree, a merge operation is triggered. In this step, the system compares data blocks in different levels and performs merge sorting based on the specified sorting field (here, the start timestamp).

[0066] Through the merge sorting algorithm, the system can effectively sort the originally unordered transaction change logs according to their start timestamps. After sorting, these log entries are reorganized into ordered files conforming to the Paimon data lake format.

[0067] Finally, the ordered files obtained through the above processing not only contain the correct transaction order, but also meet all the requirements of the Paimon data lake format, and can be directly used for data analysis, snapshot generation or other related applications.

[0068] This LSM Tree-based data processing method not only improves the data processing efficiency, but also ensures the consistency and order of transactions, which is of great significance for building real-time and large-scale database snapshots. At the same time, this method also fully utilizes the advantages of the Paimon data lake format, so that the generated data files can be efficiently read and analyzed in various query engines (such as Spark, Flink, etc.).

[0069] S140、Based on the transaction start timestamp or the table time field, a snapshot ID reflecting the current database state is generated using the Paimon interface.

[0070] In this embodiment, the snapshot ID is an identifier used to uniquely identify the state of the database at a certain time. By using the start timestamp of the distributed transaction or the time field information in the database table, and combining the interface provided by the Paimon data lake format, a snapshot ID reflecting the current database state can be generated.

[0071] Specifically, information is extracted from the start timestamp of the distributed transaction or the time field in the database table, and a unique snapshot ID is generated using the Paimon interface.

[0072] First, the start timestamp (StartTs) is extracted from the log of each distributed transaction. This timestamp accurately records the time point when the transaction starts, which is crucial for ensuring the consistency and order of transactions.

[0073] If the transaction start timestamp cannot be directly obtained in some cases, or a more detailed state description is needed, specific time fields can be extracted from the relevant database tables. These time fields are usually associated with specific business logic and provide more accurate time markers.

[0074] Once the time information for generating the snapshot ID is determined, the next step is to create a unique snapshot ID using the interface provided by Paimon. Paimon is a data lake format that supports real-time processing and batch processing, and its API design aims to simplify data management and query processes, including snapshot management functions.

[0075] The snapshot ID is usually composed of time information and other unique identifiers. For example, it can be a string composed of the transaction start timestamp, the database table name, and an incremental sequence number. This structure not only guarantees the uniqueness of the snapshot ID, but also facilitates subsequent management and query.

[0076] Each snapshot ID corresponds to the complete state of the database at a specific point in time. This is of great significance for data analysis, backup and recovery, and historical data query scenarios. By specifying the snapshot ID, users can quickly locate and access the desired historical data version.

[0077] By introducing the snapshot mechanism, not only can the storage space requirement be greatly reduced, but also the query efficiency can be significantly improved. Because when performing large-scale data analysis, directly operating the original data often takes a long time and is prone to errors; while using snapshots, accurate data views can be obtained in a more efficient manner.

[0078] Whenever a new transaction is committed, the system will check whether it is necessary to generate a new snapshot ID based on the latest time information. If so, it will call the relevant interfaces of Paimon to perform the snapshot generation process.

[0079] In order to ensure the correctness and uniqueness of the snapshot ID, sufficient testing is required before actual deployment. In addition, considering the performance requirement differences in different application scenarios, the snapshot generation strategy may also need to be adjusted and optimized accordingly.

[0080] In summary, the key to S140 step is how to effectively use the transaction start timestamp or time field information in the database table, and generate reliable snapshot ID with the powerful interface capability of Paimon, so as to realize the effective management and query of the database state. This process not only improves the flexibility and efficiency of data processing, but also provides a solid foundation for subsequent data analysis.

[0081] S150, querying the ordered lake format file and the snapshot ID using a standard data lake reader.

[0082] In this implementation, the standard data lake readers include Flink, Spark, and Doris.

[0083] With the ordered lake format files and snapshot IDs generated through the preceding steps, standard data lake readers such as Flink, Spark, and Doris can be used to efficiently query these data.

[0084] Flink: A stream processing framework that also supports batch processing, known for its low latency and high throughput. It seamlessly integrates with Paimon's data lake format, providing real-time data analysis capabilities.

[0085] Spark: A widely used distributed computing system that supports SQL queries, stream processing, and machine learning for advanced analysis functions. Spark is also tightly integrated with Paimon, making it easy for users to access and process large-scale datasets.

[0086] Doris: A modern MPP (Massively Parallel Processing) database designed for efficient OLAP (Online Analytical Processing). Doris' support for Paimon makes it an ideal choice for querying and analyzing Paimon's data lake format.

[0087] First, the generated ordered lake format files need to be loaded into the chosen data lake reader. This typically involves specifying the correct path and file format (in this case, Paimon format) so that the reader knows how to parse and process the data.

[0088] Once the data is correctly loaded, it can be queried using SQL statements or other programming interfaces. For example, in Spark SQL, you can write a query like SELECT * FROM lake_table WHERE snapshot_id='specific_snapshot_id' to retrieve data corresponding to a specific snapshot ID.

[0089] Each snapshot has a unique identifier (snapshot ID) that represents the state of the database at a certain point in time. To query the data state at a specific time point, the corresponding snapshot ID must be accurately identified.

[0090] With the snapshot ID, you can directly access that snapshot through the data lake reader. For example, in Flink, you might need to set up a job to point to the specific snapshot ID and start processing the data stream from there.

[0091] Given the need for large-scale data processing, certain strategies should be employed to optimize query performance. For example, pre-building indexes, partitioning reasonably, and parallelizing processing are all effective methods to improve efficiency.

[0092] As data originates from a distributed environment, it is crucial to ensure fault tolerance during query processes. Modern data lake readers all provide built-in failure recovery mechanisms to ensure that queries can be completed even in the presence of network instability or hardware failures.

[0093] Choosing the right data lake reader not only improves the performance of current query tasks but also leaves enough room for future expansion. Whether adding new data sources or introducing more complex analysis models, it can be relatively easy to implement.

[0094] In summary, the S150 step aims to demonstrate how to effectively query the ordered lake format files and specific snapshot IDs generated by the previous steps using standard data lake readers such as Flink, Spark, and Doris. This step not only enables precise access to historical data states but also provides strong support for subsequent data analysis and business decision-making. In this way, enterprises can quickly extract valuable information from massive amounts of data, making more informed decisions.

[0095] The method of the embodiment obtains the change log of the distributed database through an RPC request, assembles transactions, and uses the Paimon data lake LSM Tree data structure to sort and generate Paimon data lake format data to restore database operations and generate snapshot data. Finally, these snapshot data are queried by a query engine such as Spark, Flink, etc., to realize real-time and large-scale database snapshots. A distributed database is a database system that stores data on multiple physical nodes, which are usually connected together through a network. Unlike traditional centralized databases, distributed databases provide higher scalability, availability, and performance by distributing data across multiple nodes. Transactions are a method for ensuring data consistency in databases, usually implemented based on the Percolator algorithm. The Paimon data lake format is a data lake format that supports the construction of real-time lake warehouse architecture using Flink and Spark, can handle both streaming and batch processing operations, and innovatively combines the data lake format with the LSM (Log-Structured Merge) structure, introducing real-time streaming updates into the lake warehouse architecture.

[0096] The method of the embodiment directly converts the change log of the distributed database into the data lake data format, and uses the snapshot method of the data lake to snapshot the data. Specifically, by reading the change log of the distributed database, transaction assembly is performed, and sorting, data generation and snapshot generation are performed using the LSM Tree lake format. The produced lake format file can be directly used by the lake format reader for data analysis, data backup, data cleaning, etc. First, the data reader reads the change log of the distributed database through the RPC protocol, and one distributed database node corresponds to one reader. The change log is composed of multiple fields such as field Key, Logtype, Optype, startTs, CommitTs, Value, OldValue, etc., wherein the key is the unique key of the database log; the logtype includes the log types composed of prewrite, rollback, committed; the optype is composed of Put, Delete operation type; startTs is the start time, commitTs is the commit time, value is the modified value, and OldValue is the value before modification. Then, the transaction assembler receives data from multiple distributed database nodes, and assembles the data of the same transaction into a data list. Since the data may come from different readers, the data in the list may be out of order. By extracting Key+StartTs to generate Mid, and according to Logtype to decide whether to write to cache or match all change logs of the current Mid from the cache to complete transaction assembly and write to the sorter.

[0097] After the transaction sorter receives the out-of-order transactions, it uses the distributed transaction StartTs as the sorting key. After the out-of-order transactions are written to the transaction sorter, the transaction sorter writes a transaction change log in the Paimon data lake format to the Level 0 layer of the LSM Tree and specifies the sorting field as StartTs. This triggers a merge of the LSM Tree, which sorts the out-of-order data using the merge sort in the LSM Tree merge process in the Paimon data lake format and assembles it into a lake format file, which is then written to the storage. The snapshot generator extracts the snapshot ID from the distributed transaction StartTs or the time field in the distributed database table to generate the snapshot. The snapshot is generated using the snapshot interface of Paimon itself. If the out-of-order data is not sorted correctly, for example, the order of operations on the database is Delete, Insert, but after being out of order, it becomes Insert, Delete, then the data read by the reader is not accurate. Finally, the lake format file generated by the transaction sorter and the snapshot file generated by the snapshot generator can be directly provided to a standard data lake reader (such as Flink, Spark, Doris) for reading according to the Paimon data format, from which the data of the distributed database can be read, and can also be provided to the downstream system for data reading or analysis. Open source tools such as Flink, Spark, Doris, etc. can be used to read these data.

[0098] The embodiment proposes a complete method of generating Paimon data lake format files and snapshot files using distributed database change logs, sorting the out-of-order distributed transactions using the Paimon data lake format LSM Tree, extracting the time field in the transaction change log of the change log, generating Paimon snapshot data, and providing it to the downstream lake format reader; combining the out-of-order data of the distributed database change log, using the data merge sorting capability of the Paimon data lake format LSM Tree, sorting the data in the process of generating the LSM Tree merge in the Paimon data lake, and triggering the submission and Compact of the LSM Tree once for each transaction submission, which can ensure that the downstream can be accessed after the data processing is completed, and guarantee the real-time performance of the database snapshot; it is proposed to use the transaction StartTs or the time field of the distributed database change log to generate the Paimon data lake format to generate the snapshot logic, innovatively generate the snapshot of the incremental data, and greatly save the storage space.

[0099] The method utilizes the snapshot technology of the data lake to take snapshots of the data. Specifically, the present scheme involves reading the change logs of the distributed database, assembling transactions, and sorting these change logs using the LSM Tree structure to generate ordered data lake format files, so that the generated lake format files can be directly used by standard data lake readers, suitable for scenarios such as data analysis, data backup, and data cleaning. In addition, the method of the present embodiment emphasizes the ability to sort the possible out-of-order of distributed transactions using the LSM Tree feature of the Paimon data lake format. Further, the method of the present embodiment also proposes a method of implementing data sorting in the LSM Tree merging process in generating the Paimon data lake, in combination with the out-of-order problem that may exist in the distributed database change logs, and the LSM Tree commit and Compact operations are triggered once per transaction commit, ensuring that the data is available for downstream systems immediately after processing is completed, guaranteeing the real-time nature of the database snapshot. Finally, the method of the present embodiment proposes using the transaction start timestamp (StartTs) in the distributed database change logs or the time field within the table to generate the snapshot logic of the Paimon data lake format, which not only greatly saves storage space, but also provides a new approach to incremental data snapshots, providing convenience for subsequent data processing. The entire process ensures that the data migration from the distributed database to the data lake is both simplified and efficient, while guaranteeing the consistency and integrity of the data during transmission between different systems.

[0100] The above-mentioned distributed database incremental snapshot method obtains the change logs of the distributed database nodes and extracts the transaction key information therein, then assembles the change logs by transactions to obtain ordered results, uses the transaction start timestamp as the sorting key and performs merge sorting on the assembled results through the LSM tree structure to generate ordered lake format files, generates a snapshot ID reflecting the current database state based on the transaction start timestamp or the table time field using the Paimon interface, and finally queries the ordered lake format files and snapshot ID using a standard data lake reader, which not only simplifies the data migration process from the distributed database to the data lake, but also greatly improves the efficiency and reliability of the entire process. In addition, this method ensures the consistency and integrity of data transmission between different systems, as it relies on accurate timestamp sorting and efficient transaction management mechanisms to ensure that even out-of-order change logs can be correctly sorted and applied to generate accurate snapshots, thereby supporting real-time data analysis and decision-making.

[0101] Figure 4 is a schematic block diagram of a distributed database incremental snapshot device 300 provided by an embodiment of the present application. As shown in Figure 4As shown, corresponding to the above-described distributed database incremental snapshot method, the present invention also provides a distributed database incremental snapshot apparatus 300. This distributed database incremental snapshot apparatus 300 includes a unit for executing the above-described distributed database incremental snapshot method, and the apparatus can be configured in a server. Specifically, please refer to... Figure 4 The distributed database incremental snapshot device 300 includes an acquisition unit 301, an assembly unit 302, a sorting unit 303, a snapshot unit 304, and a query unit 305.

[0102] The acquisition unit 301 is used to acquire change logs from distributed database nodes, wherein the change logs include key transaction information; the assembly unit 302 is used to assemble the change logs into transactions to obtain an assembly result; the sorting unit 303 is used to use the start timestamp of the transaction as the sorting key and to merge and sort the assembly result through a log structure merge tree structure to generate an ordered lake format file; the snapshot unit 304 is used to generate a snapshot ID reflecting the current database state using the Paimon interface based on the transaction start timestamp or the table time field; and the query unit 305 is used to query the ordered lake format file and the snapshot ID using a standard data lake reader.

[0103] In one embodiment, the acquisition unit 301 is used to acquire change logs from distributed database nodes via the RPC protocol.

[0104] In one embodiment, the assembly unit 302 is used to extract a unique identifier and a start timestamp for each change log to generate a Mid value, and to classify the logs according to their types to obtain an assembly result.

[0105] In one embodiment, the assembly unit 302 is further configured to: write the change log to the cache when the log type is prewrite; delete the change log with the corresponding Mid value from the cache when the log type is rollback; and match and assemble all change logs belonging to the same transaction from the cache according to the Mid value when the log type is committed, forming a data list containing out-of-order change logs.

[0106] In one embodiment, such as Figure 5 As shown, the sorting unit 303 includes a setting subunit 3031, a writing subunit 3032, and a sorting and merging subunit 3033.

[0107] The setting sub-unit 3031 is configured to set the start timestamp of the distributed transaction as a sorting key; the writing sub-unit 3032 is configured to write the received disordered transaction change log into the 0 layer of the log-structured merge tree in the Paimon data lake format, and explicitly specify the start timestamp as a sorting field; and the sorting and merging sub-unit 3033 is configured to trigger a merge of the log-structured merge tree when data is written into the 0 layer, and in the merging process, the sorting and merging sub-unit 3033 sorts the assembly result according to the start timestamp, and reassembles the assembly result into a file in the data lake format to obtain an ordered lake format file.

[0108] In an embodiment, the snapshot unit 304 is configured to extract information from the start timestamp of the distributed transaction or a time field in a database table, and generate a unique snapshot ID using a Paimon interface.

[0109] It should be noted that the specific implementation process of the distributed database incremental snapshot device 300 and each unit can be clearly understood by those skilled in the art, and can refer to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, details are not repeated here.

[0110] The distributed database incremental snapshot device 300 described above can be implemented in the form of a computer program, which can run on a computer device as shown in the computer device. Figure 6

[0111] Please refer to Figure 6 , Figure 6 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 can be a server, wherein the server can be a stand-alone server or a server cluster composed of multiple servers.

[0112] Referring to Figure 6 , the computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501, wherein the memory can include a non-volatile storage medium 503 and an internal memory 504.

[0113] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which when executed, can cause the processor 502 to perform a distributed database incremental snapshot method.

[0114] The processor 502 is configured to provide computing and control capabilities to support the operation of the entire computer device 500.

[0115] ​The memory 504 provides an environment for running the computer program 5032 in the nonvolatile storage medium 503, which, when executed by the processor 502, can cause the processor 502 to perform a distributed database incremental snapshot method.

[0116] The network interface 505 is used for network communication with other devices. Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 500 to which the scheme of the present application is applied. The specific computer device 500 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] The processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps: obtain a change log from a distributed database node, wherein the change log includes transaction key information; perform transaction assembly on the change log to obtain an assembly result; perform merge sorting on the assembly result by a log-structured merge tree structure using a start timestamp of a transaction as a sorting key to generate an ordered lake format file; generate a snapshot ID reflecting a current database state based on a transaction start timestamp or a table internal time field using a Paimon interface; and query the ordered lake format file and the snapshot ID using a standard data lake reader.

[0118] The transaction key information includes a unique identifier, a log type of prewrite, rollback or commit, an operation type, a start timestamp, a commit timestamp, a modified value and a modified value.

[0119] The standard data lake reader includes Flink, Spark and Doris.

[0120] In an embodiment, the processor 502, when implementing the step of obtaining a change log from a distributed database node, specifically implements the following steps: Obtain a change log from a distributed database node through an RPC protocol.

[0121] In an embodiment, the processor 502, when implementing the step of performing transaction assembly on the change log to obtain an assembly result, specifically implements the following steps: For each change log, extract a unique identifier and a start timestamp to generate a Mid value, and perform classification processing according to a log type to obtain an assembly result.

[0122] In an embodiment, the processor 502, when implementing the step of performing classification processing according to a log type, specifically implements the following steps: When the log type is prewrite, the change log is written into the cache; when the log type is rollback, the change log corresponding to the Mid value is deleted from the cache; when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value to form a data list containing out-of-order change logs.

[0123] In an embodiment, when the processor 502 implements the step of sorting the assembly results by using the start timestamp of the transaction as the sorting key through the log-structured merge tree structure to generate an ordered lake format file, the following steps are implemented: The start timestamp of the distributed transaction is set as the sorting key; the received out-of-order transaction change logs are written into the 0 layer of the log-structured merge tree in the Paimon data lake format, and the start timestamp is explicitly specified as the sorting field; when the data is written into the 0 layer, a merge of the log-structured merge tree is triggered, and in the merging process, the sorting sorts the assembly results according to the start timestamp, and reassembles the assembly results into a file conforming to the data lake format to obtain an ordered lake format file.

[0124] In an embodiment, when the processor 502 implements the step of generating a snapshot ID reflecting the current database state based on the start timestamp of the transaction or the time field in the table using the Paimon interface, the following steps are implemented: Information is extracted from the start timestamp of the distributed transaction or the time field in the database table, and a unique snapshot ID is generated using the Paimon interface.

[0125] It should be understood that, in the embodiments of the present application, the processor 502 can be a central processing unit (CPU), and the processor 502 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0126] Those skilled in the art can understand that all or part of the processes in the method of implementing the above embodiments can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments of the method.

[0127] Therefore, the present application also provides a storage medium. The storage medium can be a computer readable storage medium. The storage medium stores a computer program, wherein the computer program is executed by a processor to make the processor execute the following steps: obtaining a change log from a distributed database node, wherein the change log includes transaction key information; transaction assembling the change log to obtain an assembly result; using the start timestamp of the transaction as the sorting key, performing merge sorting on the assembly result through a log structure merge tree structure to generate an ordered lake format file; using the Paimon interface based on the transaction start timestamp or the table internal time field to generate a snapshot ID reflecting the current database state; using a standard data lake reader to query the ordered lake format file and the snapshot ID.

[0128] The transaction key information includes a unique identifier, a pre-write, a rollback or a committed log type, an operation type, a start timestamp, a commit timestamp, a modified value and a modified value.

[0129] The standard data lake reader includes Flink, Spark and Doris.

[0130] In an embodiment, when the processor executes the computer program to implement the step of obtaining the change log from the distributed database node, the following steps are specifically implemented: The change log from the distributed database node is obtained through an RPC protocol.

[0131] In an embodiment, when the processor executes the computer program to implement the step of transaction assembling the change log to obtain an assembly result, the following steps are specifically implemented: For each change log, the unique identifier and the start timestamp are extracted to generate a Mid value, and the log type is classified to obtain an assembly result.

[0132] In an embodiment, when the processor executes the computer program to implement the step of classifying processing according to the log type, the following steps are specifically implemented: When the log type is prewrite, the change log is written into the cache; when the log type is rollback, the change log corresponding to the Mid value is deleted from the cache; when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value to form a data list containing out-of-order change logs.

[0133] In an embodiment, when the processor executes the computer program to implement the step of using the start timestamp of a transaction as a sorting key, performing merge sort on the assembly result through a log-structured merge tree structure to generate an ordered lake format file, the following steps are implemented: The start timestamp of a distributed transaction is set as a sorting key; the received out-of-order transaction change log is written into layer 0 of a log-structured merge tree in Paimon data lake format, and the start timestamp is explicitly specified as a sorting field; when data is written into layer 0, a merge of the log-structured merge tree is triggered, and in the merge process, the assembly result is sorted according to the start timestamp, and the assembly result is reassembled into a file conforming to the data lake format to obtain an ordered lake format file.

[0134] In an embodiment, when the processor executes the computer program to implement the step of generating a snapshot ID reflecting the current database state based on the transaction start timestamp or the table internal time field using the Paimon interface, the following steps are implemented: Generating a snapshot ID reflecting the current database state based on the transaction start timestamp or the table internal time field using the Paimon interface The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer readable storage media that can store program codes.

[0135] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0136] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the embodiments of the apparatus described above are merely schematic. For example, the division of the units is merely a logical function division. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In a possible implementation process, the steps of the method disclosed by the present application can be implemented by using a program. The program can be stored in a computer readable storage medium, for example, a computer disk, a compact disc, a compact disc, a Blu-ray disc, a magnetic disk, a floppy disk, a hard disk, a magnetic tape, an optical disc, a read-only memory (ROM), a flash memory, a portable memory card, a computer network, or the like.

[0137] The steps in the method embodiments of the present application can be adjusted, combined and deleted in sequence according to actual needs. The units in the device embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0138] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art that makes a contribution, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0139] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A distributed database incremental snapshot method, characterized in that, include: Obtain change logs from distributed database nodes, wherein the change logs include key transaction information; The change logs are then assembled using transactions to obtain the assembly result. Using the start timestamp of the transaction as the sort key, the assembled result is merged and sorted through a log structure merge tree structure to generate an ordered lake format file; The Paimon interface is used to generate a snapshot ID that reflects the current database state, based on the transaction start timestamp or the time field in the table. Use a standard data lake reader to query the ordered lake format file and the snapshot ID.

2. The distributed database incremental snapshot method according to claim 1, characterized in that, The process of obtaining change logs from distributed database nodes includes: Change logs from distributed database nodes are obtained via the RPC protocol.

3. The distributed database incremental snapshot method according to claim 1, characterized in that, The key information of the transaction includes a unique identifier, write-ahead, rollback, or committed log type, operation type, start timestamp, commit timestamp, modified value, and unmodified value.

4. The distributed database incremental snapshot method according to claim 1, characterized in that, The step of assembling the change log into a transaction to obtain the assembly result includes: For each change log entry, a unique identifier and start timestamp are extracted to generate a Mid value, and the log entries are categorized according to their type to obtain the assembly result.

5. The distributed database incremental snapshot method according to claim 4, characterized in that, The classification process based on log type includes: When the log type is prewrite, the change log is written to the cache; when the log type is rollback, the change log with the corresponding Mid value is deleted from the cache; when the log type is committed, all change logs belonging to the same transaction are matched and assembled from the cache according to the Mid value to form a data list containing out-of-order change logs.

6. The distributed database incremental snapshot method according to claim 1, characterized in that, The process of using the start timestamp of a transaction as the sorting key and merging and sorting the assembled results through a log structure merge tree structure to generate an ordered lake format file includes: Set the start timestamp of the distributed transaction as the sort key; The received out-of-order transaction change logs are written to level 0 of the log structure merge tree in Paimon data lake format, and the start timestamp is explicitly specified as the sorting field. When data is written to level 0, a merge of the log structure merge tree is triggered. During the merge process, merge sort sorts the assembled results according to the start timestamp and reassembles the assembled results into a file that conforms to the data lake format to obtain an ordered lake format file.

7. The distributed database incremental snapshot method according to claim 1, characterized in that, The snapshot ID, generated using the Paimon interface based on the transaction start timestamp or the table time field, reflecting the current database state, includes: Extract information from the start timestamp of a distributed transaction or the time field in a database table, and use the Paimon interface to generate a unique snapshot ID.

8. The distributed database incremental snapshot method according to claim 1, characterized in that, The standard data lake readers include Flink, Spark, and Doris.

9. A distributed database incremental snapshot device, characterized in that, include: The acquisition unit is used to acquire change logs from distributed database nodes, wherein the change logs include key transaction information; An assembly unit is used to perform transaction assembly on the change log to obtain an assembly result; The sorting unit is used to merge and sort the assembled results using the start timestamp of the transaction as the sorting key and through the log structure merge tree structure to generate an ordered lake format file. Snapshot unit, used to generate a snapshot ID reflecting the current database state using the Paimon interface based on the transaction start timestamp or the table time field; The query unit is used to query the ordered lake format file and the snapshot ID using a standard data lake reader.

10. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 8.

Images