A database consistency backup method, device, equipment, medium and product

Abstract

The application discloses a database consistency backup method, device, equipment, medium and product. The database consistency backup method comprises the following steps: in response to receiving a backup instruction, waiting for a column storage engine to complete a persistent operation on a data batch until a time point of receiving the backup instruction; triggering a metadata server to execute metadata snapshot backup, and requesting a current global transaction view from a global transaction manager to obtain a global transaction snapshot point; instructing the column storage engine to backup the column storage data which has been persisted, and instructing each row storage engine to execute row storage data backup based on the global transaction snapshot point; and through the global transaction manager, pulling an active transaction log according to a set collection period, and generating a backup list based on a backup path of the row storage data, a backup path of the column storage data, metadata snapshot and the active transaction log. The technical scheme of the embodiment of the application can guarantee global data consistency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a database consistency backup method, apparatus, device, medium, and product. Background Technology

[0002] With the continuous growth of enterprise data scale and the increasing intelligence of business, the Hybrid Transactional / Analytical Processing (HTAP) database architecture, which can simultaneously carry both online transaction processing and online analytical processing workloads, has been increasingly widely used.

[0003] A typical HTAP database system usually includes a row storage engine for transaction processing and a column storage engine for data analysis. The two types of engines work together in the same system to handle high-frequency read / write and complex query needs, respectively.

[0004] However, this dual-engine architecture, with data potentially distributed across different nodes, presents new technical challenges for database backup and recovery. Traditional backup methods are mostly designed for a single storage engine, making it difficult to unify row-level data, column-level data, metadata, and transaction states to a globally consistent logical point in time during backup. If backup data lacks consistency, recovery may result in data logic errors, transaction interruptions, or distorted analysis results, failing to meet the stringent data integrity requirements of critical industries such as finance and telecommunications. Current technologies often employ business write-stop or long-term locking to forcibly align backup points to ensure row-to-column consistency in HTAP databases. This leads to prolonged service unavailability, severely impacting business continuity. Summary of the Invention

[0005] This invention provides a database consistency backup method, apparatus, device, medium, and product to solve the problem of lack of consistency in the backup of row-stored and column-stored data in HTAP databases.

[0006] According to one aspect of the present invention, a database consistency backup method is provided, characterized in that it includes: Upon receiving a backup command, wait for the columnar storage engine to complete the persistence operation on the batch of data that has been written to memory but not yet persisted up to the time the backup command was received; Trigger the metadata server to perform a metadata snapshot backup and request the current global transaction view from the global transaction manager to obtain the global transaction snapshot point; A backup instruction is sent to the column storage engine in the distributed database, instructing the column storage engine to back up the persisted column storage data. Based on the global transaction snapshot point, a backup instruction is sent to the row storage engine of each data node in the distributed database, instructing each row storage engine to perform row storage data backup based on the global transaction snapshot point. The global transaction manager retrieves active transaction logs from the moment the global transaction snapshot is obtained until the row storage backup is completed, according to the set collection period. Based on the backup path of the row storage data, the backup path of the column storage data, the metadata snapshot, and the active transaction logs, a backup list is generated.

[0007] According to another aspect of the present invention, a database consistency backup apparatus is provided, comprising: The persistence wait module is used to wait for the columnar storage engine to complete the persistence operation of the batch of data that has been written to memory but not yet persisted up to the time the backup command is received in response to the received backup command. The global transaction snapshot point acquisition module is used to trigger the metadata server to perform metadata snapshot backup and request the current global transaction view from the global transaction manager to obtain the global transaction snapshot point; The data backup module is used to send backup instructions to the column storage engine in the distributed database, instructing the column storage engine to back up the persistent column storage data, and to send backup instructions to the row storage engine of each data node in the distributed database based on the global transaction snapshot point, instructing each row storage engine to perform row storage data backup based on the global transaction snapshot point. The backup list generation module is used to pull active transaction logs from the moment the global transaction snapshot is obtained until the row storage backup is completed, according to the set collection period through the global transaction manager, and generate a backup list based on the backup path of the row storage data, the backup path of the column storage data, the metadata snapshot, and the active transaction logs.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the database consistency backup method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the database consistency backup method according to any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the database consistency backup method of any embodiment of the present invention.

[0011] The technical solution of this invention, in response to receiving a backup instruction, waits for the column-oriented storage engine to complete the persistence operation of the batch of data that has been written to memory but not yet persisted up to the time of receiving the backup instruction, triggers the metadata server to perform metadata snapshot backup, requests the current global transaction view from the global transaction manager to obtain the global transaction snapshot point, sends a backup instruction to the column-oriented storage engine in the distributed database, instructing the column-oriented storage engine to back up the persisted column-oriented storage data, and, based on the global transaction snapshot point, sends a backup instruction to the row-oriented storage engine of each data node in the distributed database, instructing each row-oriented storage engine to perform row-oriented storage data backup based on the global transaction snapshot point. The global transaction manager pulls the active transaction log from the time the global transaction snapshot point is obtained to the completion of the row-oriented storage backup according to a set collection period, and generates a backup list based on the backup path of the row-oriented storage data, the backup path of the column-oriented storage data, the metadata snapshot, and the active transaction log. This can ensure global data consistency while minimizing the impact on database business.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a database consistency backup method provided according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of a database consistency backup method provided according to Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a database consistency backup device according to Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the database consistency backup method of this invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] Example 1 Figure 1 The flowchart of a database consistency backup method is provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of performing consistent backup of row-store data and column-store data in HTAP database. The method can be applied in a distributed HTAP database system, which includes a cluster management module, a global transaction manager, a metadata server, and row-store engines and column-store engines in each data node.

[0018] This method can be executed by a database consistency backup device, which can be implemented in hardware and / or software and can be configured in various general-purpose computing devices. For example... Figure 1 As shown, the method includes: S110. In response to receiving a backup instruction, wait for the columnar storage engine to complete the persistence operation of the batch of data that has been written to memory but not yet persisted up to the time the backup instruction is received.

[0019] A backup command is an instruction issued by the upper-level management tool to the cluster management module, instructing the cluster management module to perform a consistent database backup. The backup command includes the backup type (full / incremental), the target storage path, and a list of data nodes to be backed up.

[0020] In this embodiment of the invention, after receiving a backup command from the upper-layer management tool, the cluster management module parses the backup parameters (including backup type, target storage path, etc.) carried in the backup command and records the time of receipt of the backup command. Then, the cluster management module waits for the columnar storage engine to complete the persistence operation on batches of data that have been written to memory but not yet persisted up to the time the backup command was received.

[0021] Specifically, after receiving a backup command, the cluster management module enters a passive waiting state, continuously calling the columnar storage engine's status query interface at a preset polling interval (e.g., every 100 milliseconds) to obtain the current persistence progress timestamp of the columnar storage engine. This timestamp reflects the latest data commit time contained in all columnar data files currently written to disk by the columnar storage engine. The persistence progress timestamp is maintained internally by the columnar storage engine, which independently executes persistence scheduling according to its own set time period: when the amount of data in the memory buffer reaches a preset threshold or when a preset duration has elapsed since the last persistence operation, the columnar storage engine's background merge thread merges the data in the memory buffer into batches and writes them to the columnar data files on disk, updating the persistence progress timestamp upon completion of the write operation.

[0022] During each polling round, the cluster management module compares the current persistence progress timestamp returned by the columnar storage engine with the time the backup command was received. If the current persistence progress timestamp is not earlier than the time the backup command was received, it indicates that all data batches written to the columnar storage engine's memory up to the time the backup command was received have completed the persistence operation, and the cluster management module confirms and waits for completion.

[0023] Throughout the waiting process, the cluster management module does not interfere with the columnar storage engine's persistence scheduling strategy, nor does it send forced flush commands. The columnar storage engine performs persistence operations entirely according to its own runtime cycle, while the cluster management module only performs status monitoring and progress determination.

[0024] S120. Trigger the metadata server to perform metadata snapshot backup and request the current global transaction view from the global transaction manager to obtain the global transaction snapshot point.

[0025] A metadata snapshot is a frozen copy of the overall state of the database system's metadata at a specific point in time. Metadata includes the table structure definition (Schema), column attributes, primary key and index definitions, partitioning strategies, data sharding distribution topology information on distributed nodes, constraints, and storage format parameters, etc.

[0026] Metadata snapshots are used to restore data files to the metadata version at the time the snapshot was created when data is subsequently restored, ensuring that the content of the data files matches the version of the table structure.

[0027] A global transaction snapshot is a set of transaction-level baseline information used in a distributed database system to identify whether the database is in a globally consistent state at a specific logical moment. Specifically, a global transaction snapshot includes a maximum global transaction identifier and a list of active transaction identifiers. The maximum global transaction identifier is the highest value among the transaction numbers that have been committed and completed on all data shards at the time the backup command is initiated; the list of active transaction identifiers includes the transaction numbers and their respective start times for all transactions in the preparation phase that have not yet been committed or rolled back at the time the backup command is initiated. Global transaction snapshots provide a unified consistency baseline for subsequent row-level physical backups and logical backups.

[0028] In this embodiment of the invention, the cluster management module sends a metadata snapshot instruction to the metadata service. Upon receiving the instruction, the metadata service performs a snapshot read operation on the entire metadata set, including the current table structure information, partition distribution information, index definition information, and data sharding topology information, effectively freezing the current metadata version at the transaction consistency level. The frozen metadata snapshot is then written in serialized format to the metadata subdirectory under the target storage path, generating a metadata snapshot file. After the metadata snapshot file is written, the storage path of the original snapshot file is reported to the cluster management module.

[0029] The cluster management module sends a global transaction view request to the global transaction manager, requesting the current global transaction view. In response, the global transaction manager queries its currently maintained sequence of globally committed transaction identifiers, selects the transaction identifier with the largest value from the sequence as the maximum global transaction identifier. Then, it obtains all transaction identifiers currently in the preparation phase but not yet committed, along with their corresponding start times, forming a list of active transaction identifiers.

[0030] Furthermore, the global transaction manager selects all transactions from the list of active transaction identifiers whose start time is no later than the commit time corresponding to the largest global transaction identifier, forming a set of transactions to be waited for. It continuously monitors the state changes of each transaction in the set until all transactions in the set have changed to a committed or rolled-back state. Finally, the global transaction manager associates the largest global transaction identifier with the list of active transaction identifiers and the final state (commit or rollback) of each transaction in the set to form a global transaction snapshot point, and returns the global transaction snapshot point to the cluster management module.

[0031] S130. Send a backup instruction to the column storage engine in the distributed database, instructing the column storage engine to back up the persisted column storage data, and send a backup instruction to the row storage engine of each data node in the distributed database based on the global transaction snapshot point, instructing each row storage engine to perform row storage data backup based on the global transaction snapshot point.

[0032] In this embodiment of the invention, the cluster management module sends a backup command to the columnar storage engine in the distributed database, instructing the columnar storage engine to back up the persisted columnar storage data. Specifically, it obtains the distribution list of each columnar data file on the disk and the size information of each file in the columnar storage engine, and starts an asynchronous copy thread pool according to preset parallelism parameters. Each worker thread reads the columnar data file in parallel according to the distribution list, and writes the read data to the corresponding shard file under the target storage path. After completing the writing of its respective shard, each worker thread reports to the cluster management module. When any thread exits abnormally, the cluster management module records the shard start offset and shard length corresponding to that thread, releases the partially written shard file, and reallocates the shard start offset and shard length to idle threads, which then resume copying from the shard start offset.

[0033] Furthermore, based on the global transaction snapshot point, backup instructions are sent to the row storage engines of each data node in the distributed database, instructing each row storage engine to perform row storage data backup based on the global transaction snapshot point. Specifically, the cluster management module encapsulates the maximum global transaction commit identifier in the global transaction snapshot point into a physical backup instruction and sends it to the row storage engines of each data node. Each data node's row storage engine, using the maximum global transaction commit identifier as the snapshot version parameter, traverses all data shards on that node, creates a snapshot for the storage file corresponding to the valid data version whose commit timestamp is no later than the maximum global transaction commit identifier, and copies the snapshot file to the corresponding node's physical backup subdirectory under the target storage directory. After copying, the storage path is reported to the cluster management module.

[0034] Furthermore, the row storage engine identifies active transaction identifiers with transaction commit identifiers less than the maximum global transaction identifier from among the active transaction identifiers in the global transaction snapshot point as target transaction identifiers, and rolls back the row storage data written by the target transaction corresponding to the target transaction identifier based on the final state of the target transaction identifier in the global transaction snapshot point.

[0035] In addition, the row storage engine of each data node also backs up the transaction logs of the distributed database.

[0036] Optionally, the backup operation of columnar storage data can be performed in parallel with the backup operation of row storage data of each row storage engine.

[0037] In this optional embodiment, the backup operation of column-oriented data is executed in parallel with the backup operation of row-oriented data for each row-oriented storage engine. Employing a parallel backup mechanism minimizes the impact on online database services and achieves consistent database backup.

[0038] S140. Using the global transaction manager, retrieve the active transaction logs from the moment the global transaction snapshot point is obtained until the row storage backup is completed, according to the set collection period. Generate a backup list based on the backup path of the row storage data, the backup path of the column storage data, the metadata snapshot, and the active transaction logs.

[0039] In this embodiment of the invention, the cluster management module records the completion time of the row storage backup task as the end time, records the acquisition time of the global transaction snapshot point as the start time, and sends an interval query request carrying the start time and end time to the global transaction manager. The request includes the collection period, and the collection period is shorter than the normal log collection period of the global transaction manager.

[0040] In response to a range query request, the global transaction manager retrieves all state change entries from its transaction state change log buffer that have state change times between the aforementioned start and end times. Each state change entry contains at least a transaction identifier, its state before the change, its state after the change, and a change timestamp. Finally, the global transaction manager streams the retrieved state change entries back to the cluster management module in ascending chronological order. The cluster management module receives and temporarily stores the streamed state change entries, filtering out entries whose state has changed to committed or rolled back, and then writes them to the active transaction log.

[0041] Finally, the backup paths for row-oriented data, column-oriented data, metadata snapshots, and active transaction logs are compiled into a backup list to complete the backup task. Adding the active logs to the backup list provides the necessary context for recovering incomplete transactions, supporting recovery down to the transaction level.

[0042] The technical solution of this invention, in response to receiving a backup instruction, waits for the column-oriented storage engine to complete the persistence operation of the batch of data that has been written to memory but not yet persisted up to the time of receiving the backup instruction, triggers the metadata server to perform metadata snapshot backup, requests the current global transaction view from the global transaction manager to obtain the global transaction snapshot point, sends a backup instruction to the column-oriented storage engine in the distributed database, instructing the column-oriented storage engine to back up the persisted column-oriented storage data, and, based on the global transaction snapshot point, sends a backup instruction to the row-oriented storage engine of each data node in the distributed database, instructing each row-oriented storage engine to perform row-oriented storage data backup based on the global transaction snapshot point. The global transaction manager pulls the active transaction log from the time the global transaction snapshot point is obtained to the completion of the row-oriented storage backup according to a set collection period, and generates a backup list based on the backup path of the row-oriented storage data, the backup path of the column-oriented storage data, the metadata snapshot, and the active transaction log. This can ensure global data consistency while minimizing the impact on database business.

[0043] Example 2 Figure 2 This is a flowchart of a database consistent backup method provided in Embodiment 2 of the present invention. This embodiment further refines the above embodiment, providing specific steps for sending backup instructions to the row storage engines of each data node in the distributed database based on the global transaction snapshot point, instructing each row storage engine to perform row storage data backup based on the global transaction snapshot point. For example... Figure 2 As shown, the method includes: S210. In response to receiving a backup instruction, wait for the columnar storage engine to complete the persistence operation of the batch of data that has been written to memory but not yet persisted up to the time the backup instruction is received.

[0044] S220. Trigger the metadata server to perform metadata snapshot backup and request the current global transaction view from the global transaction manager to obtain the global transaction snapshot point.

[0045] Optionally, request the current global transaction view from the global transaction manager to obtain a global transaction snapshot point, including: A global transaction view request is sent to the global transaction manager. In response to the global transaction view request, the global transaction manager obtains the maximum number in the current global committed transaction sequence as the maximum global transaction identifier, and obtains the identifiers of transactions that are currently in the preparation stage and have not yet been committed, as well as their respective start times, to form an active transaction identifier list. The active transaction identifier list selects transactions whose start time is no later than the commit time corresponding to the maximum global transaction identifier, and sets them as a waiting transaction set. The global transaction manager is instructed to monitor the status of each transaction in the waiting transaction set until the transaction status changes to committed or rolled back. The global transaction manager associates the maximum global transaction identifier, the list of active transaction identifiers, and the final state of each transaction in the waiting transaction set as a global transaction snapshot point.

[0046] In this optional embodiment, a specific method is provided for requesting the current global transaction view from the global transaction manager to obtain a global transaction snapshot point: a global transaction view request is sent to the global transaction manager. In response, the global transaction manager obtains the largest number in the current global committed transaction sequence as the largest global transaction identifier, and obtains the identifiers of transactions currently in the preparation phase but not yet committed, along with their respective start times, forming a list of active transaction identifiers. Specifically, the global transaction manager queries its maintained global committed transaction sequence. This sequence records the transaction numbers that have been committed on all data shards in the distributed database, arranged in ascending order of number. The global transaction manager searches backward from the end of the sequence (i.e., the maximum value end) and selects the first number as the largest global transaction identifier.

[0047] Furthermore, transactions with start times no later than the commit time corresponding to the maximum global transaction identifier are selected from the list of active transaction identifiers to form a waiting transaction set. The global transaction manager is then instructed to monitor the status of each transaction in the waiting transaction set until all transaction statuses have changed to committed or rolled back. Specifically, from the list of active transaction identifiers, the start time of each transaction is compared one by one with the commit time corresponding to the maximum global transaction identifier, and all transactions with start times no later than that commit time are selected to form the waiting transaction set. The cluster management module sends a monitoring instruction to the global transaction manager, instructing the global transaction manager to continuously monitor the status changes of each transaction in the waiting transaction set. The global transaction manager monitors whether each transaction in the waiting transaction set has changed from the preparation phase to the committed or rolled-back state by subscribing to the status change events of the active transaction table or by periodic polling. When the global transaction manager confirms that all transactions in the waiting transaction set have changed to the committed or rolled-back state, it returns a confirmation of completion to the cluster management module. The global transaction manager associates the maximum global transaction identifier, the list of active transaction identifiers, and the final status of each transaction in the waiting transaction set as a global transaction snapshot point.

[0048] By taking the maximum value from the global committed transaction sequence identifiers as the maximum global transaction identifier, the precise anchoring of the column-store physical watermark and the global transaction logical watermark in terms of transaction identifier dimension is ensured. This aligns the cutoff point of the row-store physical backup with the disk placement point of the column-store data. By monitoring the final state of each transaction in the waiting transaction set and incorporating this state information into the global transaction snapshot point, accurate transaction processing basis is provided for subsequent row-store logical backups, improving data backup accuracy. The maximum global transaction identifier, the list of active transaction identifiers, and the final state of each transaction are associated and encapsulated into a global transaction snapshot point, providing a unified transaction-level consistency reference for subsequent column-store backups, row-store physical backups, and row-store logical backups. All backup tasks are executed based on the same snapshot point, ensuring the overall logical consistency of the backup data.

[0049] S230. Send a backup command to the columnar storage engine in the distributed database, instructing the columnar storage engine to back up the persisted columnar storage data.

[0050] Optionally, a backup command is sent to the column-oriented storage engine in the distributed database, instructing the column-oriented storage engine to back up the persisted column-oriented data, including: A backup command is sent to the columnar storage engine in the distributed database. The columnar storage engine initiates an asynchronous backup task and uses a multi-threaded approach to perform parallel copying or parallel export operations on the persisted columnar storage data.

[0051] In this optional embodiment, a specific method is provided for sending a backup command to the column-oriented storage engine in the distributed database, instructing the column-oriented storage engine to back up the persisted column-oriented storage data: After obtaining a global transaction snapshot point, the cluster management module sends a backup command to the column-oriented storage engine in the distributed database. The backup command includes the target storage path and parallelism parameters for this backup. Specifically, after receiving the backup command, the column-oriented storage engine starts the backup execution process asynchronously. The cluster management module polls the column-oriented storage engine's backup task execution progress through a status query interface, and determines that the column-oriented storage backup task is complete when all shards have been copied. The column-oriented storage engine obtains a distribution list of all column-oriented data files on its disk, including: the absolute path of each file, file size, disk volume, and storage medium type information. The distribution list reflects the physical layout of the full amount of column-oriented storage data to be processed in this backup on the disk. The column-oriented storage engine creates an asynchronous copy thread pool with a preset concurrency level according to the parallelism parameters. Each worker thread performs parallel copy or parallel export operations on the persisted column-oriented storage data.

[0052] Once all fragments of all column data files have been successfully written to the target storage path, the column storage engine reports the column storage backup task completion status to the cluster management module.

[0053] It should be noted that during the asynchronous backup task performed by the columnar storage engine, the normal data writing of the columnar storage engine is not affected. New data batches can continue to be written to the memory buffer and persisted to the disk according to the set time period. The newly written data files do not affect the ongoing backup task. The backup task processes the set of columnar data files that already exist when the backup command is received, which is achieved by recording a snapshot of the file list or using the snapshot function of the file system.

[0054] By combining asynchronous task mechanisms with multi-threaded parallel replication, the parallel processing capabilities of modern multi-core CPUs and the I / O bandwidth of storage systems are fully utilized. This optimizes the backup time of large-scale table data from single-threaded serial accumulation to the maximum value of multi-threaded parallel replication, thus shortening the overall backup window.

[0055] S240: Based on the global transaction snapshot point, send backup instructions to the row storage engine of each data node in the distributed database.

[0056] In this embodiment of the invention, after the cluster management module sends a backup command to the columnar storage engine, it immediately sends a row-level storage backup command to each data node in the distributed database based on the global transaction snapshot point, without waiting for the columnar storage backup to complete. The row-level storage backup command encapsulates the maximum global transaction identifier, a list of active transaction identifiers, the final state of each transaction, and the target storage path for this backup. Upon receiving the row-level storage backup command, each data node parses the above information to prepare for subsequent row-level physical backups and transaction log backups.

[0057] S250: The row storage engine of each data node responds to the backup command, performs physical backup of row storage data based on the maximum global transaction identifier and active transaction identifier in the global transaction snapshot point, and backs up the transaction log of the distributed database.

[0058] In this embodiment of the invention, after receiving the row storage backup instruction issued by the cluster management module, each data node parses out the maximum global transaction identifier in the backup instruction, uses it as a snapshot version parameter, and calls the physical snapshot interface of its own row storage engine to perform a physical backup. Specifically, the physical snapshot interface of the own row storage engine is called, using the maximum global transaction identifier as the cutoff version number, to traverse the storage files of all data shards on the own node, and copies the physical files corresponding to the row storage data written by committed transactions whose transaction commit identifiers are not greater than the maximum global transaction identifier to the corresponding node physical backup subdirectory under the target backup directory.

[0059] Because each data shard executes physical snapshots independently, for each transaction in the active transaction identifier list, it's possible that the transaction has been committed and included in the physical backup on some data shards, while on other data shards the transaction is still in the preparation stage and not included in the physical backup. If the transaction is finally committed after the backup baseline time, each shard will face an inconsistent state during recovery, with some data from the transaction present and some data missing. Therefore, it is necessary to roll back all data written by transactions in the active transaction identifier list from their respective physical backup sets, so that the data in the final physical backup set precisely ends at the maximum global transaction identifier—a deterministic point where all shards have reached consensus—and does not contain any data from transactions in an inconsistent state across shards.

[0060] Specifically, the line storage agent service removes the data written by each active transaction on each shard from the physical backup set of that shard, one by one, based on the list of active transaction identifiers recorded in the global transaction snapshot point.

[0061] Furthermore, the row storage engine copies the transaction log files to the corresponding node log backup subdirectory under the target backup directory based on the list of all currently persisted transaction log files.

[0062] By determining the cutoff point of physical backup based on the maximum global transaction identifier and combining it with the list of active transaction identifiers, a unified rollback is performed on all transaction data in a cross-shard pending state, ensuring that the final physical backup set does not contain any transaction data with inconsistent states across shards, and that the data states of each shard are fully aligned after recovery.

[0063] Optionally, in response to a backup command, the row storage engine of each data node performs a physical backup of the row storage data based on the maximum global transaction identifier and the active transaction identifier in the global transaction snapshot point, including: In response to the backup command, the row storage engine of each data node performs a physical snapshot backup of the row storage data written by committed transactions whose transaction commit identifier is not greater than the maximum global transaction identifier, based on the maximum global transaction identifier in the global transaction snapshot point. Among the active transaction identifiers in the global transaction snapshot point, the active transaction identifier whose transaction commit identifier is less than the maximum global transaction identifier is identified as the target transaction identifier, and the row storage data written by the target transaction corresponding to the target transaction identifier is rolled back according to the final state of the target transaction identifier in the global transaction snapshot point.

[0064] In this optional embodiment, a specific method is provided for the row storage engine of each data node to perform physical backup of row storage data in response to a backup command, based on the maximum global transaction identifier and active transaction identifier in the global transaction snapshot point: After receiving the row storage backup command issued by the cluster management module, the row storage engine of each data node parses out the list of maximum global transaction identifiers and active transaction identifiers contained in the global transaction snapshot point. The row storage engine uses the maximum global transaction identifier as the cutoff version number and traverses the storage files of all data shards on the local node. For each data shard, the row storage engine calls its multi-version read interface to read the physical storage files corresponding to the data records written by committed transactions whose transaction commit identifiers are not greater than the maximum global transaction identifier in that data shard. The row storage engine copies these physical storage files to the corresponding node physical backup subdirectory under the target storage path to form a physical snapshot backup set.

[0065] Furthermore, the line storage engine identifies each active transaction one by one from the list of active transaction identifiers in the global transaction snapshot. The line storage engine identifies active transactions whose commit identifiers are less than the maximum global transaction identifier from the list of active transaction identifiers in the global transaction snapshot, and marks these transactions as target transaction identifiers. For example, the maximum global transaction identifier is 5, and the list of active transaction identifiers includes transaction identifier 4. Transaction 4 is a distributed transaction spanning multiple shards. It has been committed and completed in shard 1 (transaction commit identifier is 4), but is still in the preparation phase in shard 2. At this time, the commit identifier (4) of transaction 4 is less than the maximum global transaction identifier (5), therefore transaction 4 is identified as the target transaction.

[0066] The line storage engine performs rollback operations on the physical backup sets of each data shard based on the target transaction identifier and its final state in the global transaction snapshot. For each target transaction identifier, the line storage engine retrieves the write data records of that target transaction on each data shard. Since the transaction has been committed on some shards (therefore its data already exists in the physical backup set of those shards), but not yet committed on other shards (therefore its data does not exist in the physical backup set of those shards), the line storage engine removes all data written by that transaction from the physical backup sets of all data shards, ensuring that the rolled-back physical backup sets do not contain any data from that transaction on any data shard.

[0067] For active transactions with a commit identifier less than the maximum global transaction identifier (i.e., cross-shard distributed transactions), they may have been committed on some shards but not on others, leading to inconsistencies in the inclusion of this transaction in the physical backup data of each shard. By uniformly rolling back all data of this transaction in the physical backup sets of each data shard, it is ensured that the final generated physical backup set does not contain the data of this transaction on any data shard. The data state of each shard after recovery is completely aligned, eliminating the risk of data inconsistency after recovery caused by the incomplete cross-shard distributed transaction.

[0068] S260. Using the global transaction manager, retrieve the active transaction logs from the moment the global transaction snapshot is obtained until the row storage backup is completed, according to the set collection period. Generate a backup list based on the backup path of the row storage data, the backup path of the column storage data, the metadata snapshot, and the active transaction logs.

[0069] The technical solution of this invention, in response to receiving a backup instruction, waits for the column-oriented storage engine to complete the persistence operation of the batch of data that has been written to memory but not yet persisted up to the time of receiving the backup instruction, triggers the metadata server to perform metadata snapshot backup, requests the current global transaction view from the global transaction manager to obtain the global transaction snapshot point, sends a backup instruction to the column-oriented storage engine in the distributed database, instructing the column-oriented storage engine to back up the persisted column-oriented storage data, and, based on the global transaction snapshot point, sends a backup instruction to the row-oriented storage engine of each data node in the distributed database, instructing each row-oriented storage engine to perform row-oriented storage data backup based on the global transaction snapshot point. The global transaction manager pulls the active transaction log from the time the global transaction snapshot point is obtained to the completion of the row-oriented storage backup according to a set collection period, and generates a backup list based on the backup path of the row-oriented storage data, the backup path of the column-oriented storage data, the metadata snapshot, and the active transaction log. This can ensure global data consistency while minimizing the impact on database business.

[0070] Example 3 Figure 3This is a schematic diagram of a database consistency backup device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: The persistence wait module 310 is used to wait for the columnar storage engine to complete the persistence operation of the batch of data that has been written to memory but not yet persisted up to the time of receiving the backup instruction in response to receiving the backup instruction. The global transaction snapshot point acquisition module 320 is used to trigger the metadata server to perform metadata snapshot backup and request the current global transaction view from the global transaction manager to obtain the global transaction snapshot point. The data backup module 330 is used to send backup instructions to the column storage engine in the distributed database, instructing the column storage engine to back up the persisted column storage data, and to send backup instructions to the row storage engine of each data node in the distributed database based on the global transaction snapshot point, instructing each row storage engine to perform row storage data backup based on the global transaction snapshot point. The backup list generation module 340 is used to pull active transaction logs from the time the global transaction snapshot point is obtained to the completion of row storage backup according to the set collection period through the global transaction manager, and generate a backup list based on the backup path of row storage data, the backup path of column storage data, metadata snapshot and active transaction logs.

[0071] The technical solution of this invention, in response to receiving a backup instruction, waits for the column-oriented storage engine to complete the persistence operation of the batch of data that has been written to memory but not yet persisted up to the time of receiving the backup instruction, triggers the metadata server to perform metadata snapshot backup, requests the current global transaction view from the global transaction manager to obtain the global transaction snapshot point, sends a backup instruction to the column-oriented storage engine in the distributed database, instructing the column-oriented storage engine to back up the persisted column-oriented storage data, and, based on the global transaction snapshot point, sends a backup instruction to the row-oriented storage engine of each data node in the distributed database, instructing each row-oriented storage engine to perform row-oriented storage data backup based on the global transaction snapshot point. The global transaction manager pulls the active transaction log from the time the global transaction snapshot point is obtained to the completion of the row-oriented storage backup according to a set collection period, and generates a backup list based on the backup path of the row-oriented storage data, the backup path of the column-oriented storage data, the metadata snapshot, and the active transaction log. This can ensure global data consistency while minimizing the impact on database business.

[0072] Optionally, the backup operation of columnar storage data can be performed in parallel with the backup operation of row storage data of each row storage engine.

[0073] Optional, the data backup module 330 includes: The row storage backup instruction sending unit is used to send backup instructions to the row storage engine of each data node in the distributed database based on the global transaction snapshot point; The row storage data backup unit is used by the row storage engine of each data node to respond to the backup command, perform physical backup of row storage data based on the maximum global transaction identifier and active transaction identifier in the global transaction snapshot point, and back up the transaction log of the distributed database.

[0074] Optional, row-level backup unit, specifically used for: In response to the backup command, the row storage engine of each data node performs a physical snapshot backup of the row storage data written by committed transactions whose transaction commit identifier is not greater than the maximum global transaction identifier, based on the maximum global transaction identifier in the global transaction snapshot point. Among the active transaction identifiers in the global transaction snapshot point, the active transaction identifier whose transaction commit identifier is less than the maximum global transaction identifier is identified as the target transaction identifier, and the row storage data written by the target transaction corresponding to the target transaction identifier is rolled back according to the final state of the target transaction identifier in the global transaction snapshot point.

[0075] Optional, the global transaction snapshot acquisition module 320 is specifically used for: A global transaction view request is sent to the global transaction manager. In response to the global transaction view request, the global transaction manager obtains the maximum number in the current global committed transaction sequence as the maximum global transaction identifier, and obtains the identifiers of transactions that are currently in the preparation stage and have not yet been committed, as well as their respective start times, to form an active transaction identifier list. The active transaction identifier list selects transactions whose start time is no later than the commit time corresponding to the maximum global transaction identifier, and sets them as a waiting transaction set. The global transaction manager is instructed to monitor the status of each transaction in the waiting transaction set until the transaction status changes to committed or rolled back. The global transaction manager associates the maximum global transaction identifier, the list of active transaction identifiers, and the final state of each transaction in the waiting transaction set as a global transaction snapshot point.

[0076] Optionally, the data backup module 330 also includes: The column-oriented data backup unit is used to send backup instructions to the column-oriented engine in the distributed database. The column-oriented engine initiates an asynchronous backup task and uses a multi-threaded approach to perform parallel copying or parallel export operations on the persisted column-oriented data.

[0077] The database consistency backup device provided in this embodiment of the invention can execute the database consistency backup method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0078] In the technical solution of this invention, the information collected is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of related data all comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0079] Example 4 According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0080] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0081] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0082] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0083] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as database consistent backup methods.

[0084] In some embodiments, the database consistent backup method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the database consistent backup method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the database consistent backup method by any other suitable means (e.g., by means of firmware).

[0085] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0086] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0087] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0089] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0090] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0091] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the database consistency backup method provided in any embodiment of this application. This program product and the database consistency backup method disclosed in the embodiments of this application belong to the same inventive concept, and therefore will not be described in detail here.

[0092] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0093] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A database consistent backup method, characterized in that, include: Upon receiving a backup command, wait for the columnar storage engine to complete the persistence operation on the batch of data that has been written to memory but not yet persisted up to the time the backup command was received; Trigger the metadata server to perform a metadata snapshot backup and request the current global transaction view from the global transaction manager to obtain the global transaction snapshot point; A backup instruction is sent to the column storage engine in the distributed database, instructing the column storage engine to back up the persisted column storage data. Based on the global transaction snapshot point, a backup instruction is sent to the row storage engine of each data node in the distributed database, instructing each row storage engine to perform row storage data backup based on the global transaction snapshot point. The global transaction manager retrieves active transaction logs from the moment the global transaction snapshot is obtained until the row storage backup is completed, according to the set collection period. Based on the backup path of the row storage data, the backup path of the column storage data, the metadata snapshot, and the active transaction logs, a backup list is generated.

2. The method according to claim 1, characterized in that, The backup operations for columnar storage data are executed in parallel with the backup operations for row storage data of each row storage engine.

3. The method according to claim 1, characterized in that, Based on the global transaction snapshot point, backup instructions are sent to the row storage engines of each data node in the distributed database, instructing each row storage engine to perform row storage data backup based on the global transaction snapshot point, including: Based on the global transaction snapshot point, a backup instruction is sent to the row storage engine of each data node in the distributed database; In response to backup commands, the row storage engine of each data node performs physical backup of the row storage data based on the maximum global transaction identifier and the active transaction identifier in the global transaction snapshot point, and also backs up the transaction log of the distributed database.

4. The method according to claim 3, characterized in that, In response to backup commands, the row storage engine of each data node performs a physical backup of the row storage data based on the maximum global transaction identifier and the active transaction identifier in the global transaction snapshot point, including: In response to the backup command, the row storage engine of each data node performs a physical snapshot backup of the row storage data written by committed transactions whose transaction commit identifier is not greater than the maximum global transaction identifier, based on the maximum global transaction identifier in the global transaction snapshot point. Among the active transaction identifiers in the global transaction snapshot point, the active transaction identifier whose transaction commit identifier is less than the maximum global transaction identifier is identified as the target transaction identifier, and the row storage data written by the target transaction corresponding to the target transaction identifier is rolled back according to the final state of the target transaction identifier in the global transaction snapshot point.

5. The method according to claim 1, characterized in that, Request the current global transaction view from the global transaction manager to obtain the global transaction snapshot point, including: A global transaction view request is sent to the global transaction manager. In response to the global transaction view request, the global transaction manager obtains the maximum number in the current global committed transaction sequence as the maximum global transaction identifier, and obtains the identifiers of transactions that are currently in the preparation stage and have not yet been committed, as well as their respective start times, to form an active transaction identifier list. The active transaction identifier list selects transactions whose start time is no later than the commit time corresponding to the maximum global transaction identifier, and sets them as a waiting transaction set. The global transaction manager is instructed to monitor the status of each transaction in the waiting transaction set until the transaction status changes to committed or rolled back. The global transaction manager associates the maximum global transaction identifier, the list of active transaction identifiers, and the final state of each transaction in the waiting transaction set as a global transaction snapshot point.

6. The method according to claim 1, characterized in that, Sending a backup command to the column-oriented storage engine in the distributed database, instructing the engine to back up the persisted column-oriented data, including: A backup command is sent to the columnar storage engine in the distributed database. The columnar storage engine initiates an asynchronous backup task and uses a multi-threaded approach to perform parallel copying or parallel export operations on the persisted columnar storage data.

7. A database consistency backup device, characterized in that, include: The persistence wait module is used to wait for the columnar storage engine to complete the persistence operation of the batch of data that has been written to memory but not yet persisted up to the time the backup command is received in response to the received backup command. The global transaction snapshot point acquisition module is used to trigger the metadata server to perform metadata snapshot backup and request the current global transaction view from the global transaction manager to obtain the global transaction snapshot point; The data backup module is used to send backup instructions to the column storage engine in the distributed database, instructing the column storage engine to back up the persistent column storage data, and to send backup instructions to the row storage engine of each data node in the distributed database based on the global transaction snapshot point, instructing each row storage engine to perform row storage data backup based on the global transaction snapshot point. The backup list generation module is used to pull active transaction logs from the moment the global transaction snapshot is obtained until the row storage backup is completed, according to the set collection period through the global transaction manager, and generate a backup list based on the backup path of the row storage data, the backup path of the column storage data, the metadata snapshot, and the active transaction logs.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the database consistency backup method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the database consistency backup method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the database consistent backup method according to any one of claims 1-6.

Images