Implementation methods and systems for distributed database mirror libraries

By adopting an architecture where backup nodes and mount nodes are deployed on the same machine in the distributed database, and using write-time replication technology to generate a mirror database, the business blocking problem caused by data synchronization is solved, and the generation of a fast and globally consistent mirror database is achieved, reducing storage costs and generation time.

CN123115073APending Publication Date: 2026-11-17JINZHUAN INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611337425.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-31
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies for generating mirror libraries in distributed databases can cause business blockages due to data synchronization issues, making it impossible to generate multiple mirror libraries quickly and causing performance interference in production environments.

Method used

The architecture adopts a co-location of standby nodes and mount nodes. The standby node synchronizes transaction logs and generates data snapshot volumes, while the mount node provides mirror library services, avoiding snapshot operations on the primary node and using copy-on-write technology to quickly generate snapshot volumes.

Benefits of technology

It enables the rapid generation of mirror libraries without affecting the performance of the production library, ensuring global consistency, reducing storage usage, shortening generation time, and avoiding business blockage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115073A_ABST
    Figure CN123115073A_ABST
Patent Text Reader

Abstract

This application discloses a method and system for implementing a distributed database mirror repository. The mirror repository implementation system includes a production database cluster and a mirror repository cluster. The production database cluster includes a coordinating node, a global transaction management node, and multiple data sharding nodes. Each data sharding node includes a master node and a standby node. The mirror repository cluster includes multiple mount nodes. Each standby node corresponds one-to-one with a mount node and is deployed on the same physical device. The standby nodes are used to synchronize transaction logs from the corresponding master node and perform log replay to generate data snapshot volumes. The mount nodes are used to mount the data snapshot volumes and provide mirror repository services. This application solves the technical problem in related technologies where data synchronization can cause business blockage.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of database technology, and more specifically, to a method and system for implementing a mirror library of a distributed database. Background Technology

[0002] This section is intended to provide background or context for the content set forth in the claims or specification, and the content described herein is not acknowledged as prior art simply because it is included in this section.

[0003] Mirror libraries are an important application scenario in the database field. They are used to quickly generate mirror libraries that are consistent with the production database, enabling applications to perform off-site queries, data statistics, report generation, application verification, and system implementation drills. With the widespread adoption of distributed database technology, enterprises' demand for mirror libraries is increasing. Applications have two strict requirements for mirror libraries: first, the ability to generate mirror libraries quickly, within minutes, and to be able to generate them repeatedly; second, minimal impact on production, with no blocking of the production database during mirror library generation and no performance interference during use.

[0004] Traditional centralized databases typically use disk arrays as dedicated storage devices, generating mirror databases through the array's snapshot copy function. This involves quickly creating a data snapshot of the data mounted in the production database and mounting that snapshot to a second database instance. However, this approach relies on shared disk array storage and is unsuitable for distributed databases based on local disks. Distributed databases consist of multiple servers, with data stored on the local disks of each node. Snapshot operations between these servers are asynchronous and parallel, making it impossible to guarantee that multiple nodes will complete snapshots simultaneously, thus resulting in data inconsistency. Furthermore, performing snapshot operations directly on the production database's master node requires pausing all requests to control consistency points, which can severely disrupt production services and impact business continuity. While distributed database cluster replication technologies can generate mirror databases, they cannot be quickly regenerated. When the mirror database is modified, data needs to be re-emulated, incurring significant time overhead and failing to meet the need for multiple rapid generation cycles.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a method and system for implementing a mirror library for a distributed database, so as to at least solve the technical problem that data synchronization can cause business blockage in related technologies.

[0007] According to one aspect of the embodiments of this application, a distributed database mirror repository implementation system is provided, including: a production database cluster and a mirror repository cluster. The production database cluster includes a coordinating node, a global transaction management node, and multiple data sharding nodes. Each data sharding node includes a master node and a backup node. The mirror repository cluster includes multiple mounting nodes. The backup nodes correspond one-to-one with the mounting nodes and are deployed on the same physical device. The backup nodes are used to synchronize transaction logs from the corresponding master node and perform log replay to generate data snapshot volumes. The mounting nodes are used to mount the data snapshot volumes and provide mirror repository services.

[0008] According to another aspect of the embodiments of this application, a method for implementing a mirror library of a distributed database is also provided, including: a standby node synchronizing transaction logs from the corresponding master node and performing log replay to generate a data snapshot volume; and mounting the data snapshot volume through a mounting node corresponding to the standby node to provide a mirror library service.

[0009] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above-described method when the program is run.

[0010] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the above-described method through the computer program.

[0011] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the embodiments of the methods described above.

[0012] In this embodiment, the distributed database mirror repository implementation system includes a production database cluster and a mirror repository cluster. The production database cluster includes a coordination node, a global transaction management node, and multiple data shard nodes. Each data shard node includes a primary node and a backup node. The mirror repository cluster includes multiple mounting nodes. Each backup node corresponds one-to-one with a mounting node and is deployed on the same physical device. The backup node is used to synchronize transaction logs from the corresponding primary node and perform log replay to generate a data snapshot volume. The mounting nodes are used to mount the data snapshot volume and provide mirror repository services. By limiting the execution subject of the snapshot generation operation to the backup node, and limiting the backup node to only need to synchronize transaction logs from the primary node and perform log replay to have a data copy consistent with the primary node, the snapshot operation does not need to be executed on the primary node. Therefore, the primary node does not need to pause any requests due to the snapshot operation, thus solving the technical problem of data synchronization causing business blockage in related technologies. This avoids pausing all requests when performing snapshot operations directly on the production database primary node, which would cause severe blockage to production services. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of a database mirror library generation scheme based on a magnetic array according to an embodiment of this application; Figure 2 This is a schematic diagram of a distributed database mirror library implementation system according to an embodiment of this application; Figure 3 This is a flowchart of a distributed database mirror library implementation method according to an embodiment of this application; Figure 4 This is a schematic diagram illustrating the generation of each shard snapshot controlled by gtids snapshot in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the principle of the Copy-on-Write (COW) mechanism in an embodiment of this application. Figure 6 This is a schematic diagram of an image library generation scheme based on cluster replication technology according to an embodiment of this application; Figure 7 This is a structural block diagram of a terminal according to an embodiment of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] According to one aspect of the embodiments of this application, an embodiment of a method for implementing a mirror library of a distributed database is provided.

[0017] Optionally, in this embodiment, the above-described distributed database mirror library implementation method can be applied to a distributed database mirror library implementation system as described in this application. It includes: The system comprises a production database cluster and a mirror database cluster. The production database cluster includes a Coordinator Node (CN), a Global Transaction Manager (GTM), and multiple Data Nodes (DN). Each Data Node includes a primary node and a backup node. The mirror database cluster includes multiple mount nodes. Each backup node corresponds one-to-one with a mount node and is deployed on the same physical device. The backup node is used to synchronize transaction logs from the corresponding primary node and perform log replay. The mount nodes are used to mount data snapshot volumes and provide mirror database services. By generating data snapshots on the backup nodes, concurrency control issues caused by snapshot operations on the production database primary node are avoided.

[0018] In the specific implementation process, the production database cluster serves as the master cluster of the distributed database, responsible for handling read and write requests from business applications. The coordinating node (CN) is responsible for receiving application requests, parsing SQL statements, formulating execution plans, and coordinating the completion of distributed transaction processing by each data shard node. The global transaction management node (GTM) is responsible for allocating global transaction identifiers (GTIDs), maintaining a global active transaction list, and coordinating the commit and rollback of distributed transactions to ensure the ACID properties of distributed transactions. Each data shard node (DN) stores a portion of the database data, distributed across different DNs according to sharding rules. Each DN adopts a master-slave architecture, with the master node handling read and write requests and the slave node maintaining data synchronization with the master node through transaction log replication. The mirror database cluster is physically separated from the production database cluster and does not share server equipment to avoid resource contention. The mounting nodes in the mirror database cluster are independent database instances used to mount snapshot volumes generated based on the slave node data and provide read-only or read-write mirror database services. The standby node and the mount node are co-located on the same physical server and share local storage resources. This allows the mount node to directly use copy-on-write technology to quickly generate snapshot volumes based on the standby node's data volumes without transferring data across the network.

[0019] By adopting the above technical solution and through an innovative architecture that separates the image repository from the production repository and co-locates the standby node and the mount node, at least the following technical effects are achieved: First, the mount node and the standby machine are on the same machine, which can quickly generate data snapshots using write-time replication technology without the need to transfer the full amount of data across the network; Second, generating data snapshots on the standby node and using log stream replay as a synchronization stream and synchronization point provides triggering conditions for globally consistent snapshot points, avoiding the complex problem of needing to pause all requests to control consistency points on the production repository; Third, triggering snapshot generation on the standby machine only requires stopping log replay, and the production repository can continue to provide services without interruption, resulting in zero blocking on production; Fourth, the image repository and the production repository environments are completely separated, and access to the image repository will not cause any performance impact on the production repository.

[0020] To more clearly illustrate the deployment architecture of this embodiment, please refer to... Figure 1 and Figure 2 . Figure 1 The architecture of a database mirror repository generation technology based on magnetic arrays is illustrated. For example... Figure 1As shown, in the relevant scheme, the production database consists of three servers, and the mirror database consists of three servers. Both the production database and the mirror database are connected to a shared disk array device via a Host Bus Adapter (HBA). The VLAN 1 where the production database resides and the VLAN 2 where the mirror database resides achieve data replication through the disk array's snapshot copy function. This scheme relies on shared disk array storage, with all nodes using the same disk array device. It cannot be implemented on a distributed database based on local disks, and it cannot guarantee the global consistency of snapshots across multiple nodes.

[0021] Figure 2 This application illustrates a distributed database mirror system provided in an embodiment. For example... Figure 2 As shown, the distributed database production cluster includes a coordinating node (CN), a global transaction management node (GTM) (primary), and multiple data shard master nodes (such as DN1 primary, DNn primary, etc.). The mirror database cluster is deployed on multiple servers (Server1, Server2, ..., Server n). Each server simultaneously deploys one data shard standby node (such as DN1 standby, DNn standby, etc.) and one mount node (such as DN1 mount, DNn mount, etc.). The standby node and the mount node are one-to-one and co-located on the same physical device. The standby node synchronizes transaction logs from the corresponding primary node in the production database and performs replay to maintain data synchronization with the primary node. The mount node is used to mount data snapshot volumes and provide mirror database services. Each server's local storage contains original data blocks (used by the standby node) and data snapshots (used by the mount node). The standby node continuously receives new logs for data synchronization. Through this deployment method, the mirror database is completely separated from the production database, and the standby node and the mount node are deployed on the same machine, which ensures that the performance of the production database is not affected and provides a foundation for the rapid generation of snapshots.

[0022] In some optional embodiments of this application, the coordinating node is used to respond to the snapshot creation request, obtain the global transaction identifier and the global active transaction list from the global transaction management node, and write the global transaction identifier and the global active transaction list into the snapshot transaction table of each data shard node. The snapshot transaction table is a replication table shared by each data shard node. When each backup node identifies the log position corresponding to the write operation of the snapshot transaction table during the log replay process, it triggers the data snapshot generation operation.

[0023] In practice, the snapshot transaction table (SNAPSHOT_TRANS) is a replicated table in the distributed database. A replicated table means that the data in this table is identical on every data shard node; any modification to this table will be synchronized across all shard nodes. When the coordinating node receives a request to create a distributed snapshot, for example, triggered by executing the ALTERDATABASE SNAPSHOT command, the coordinating node first queries the Global Transaction Management (GTM) node for the current list of globally active transactions—that is, a list of identifiers for all started but not yet committed distributed transactions—and simultaneously requests a Global Transaction Identifier (GTID) for its own transaction. Then, the coordinating node inserts this global transaction identifier and the list of globally active transactions as a single record into the snapshot transaction table and commits the insertion operation within the same distributed transaction. Because the snapshot transaction table is a replicated table, this insertion operation is executed on the master nodes of all data shards, generating corresponding transaction logs. Each shard master node synchronizes the transaction logs to its corresponding standby node through a log replication mechanism. During log replay, when the standby node reaches the log point corresponding to the insertion operation, it recognizes this as a snapshot trigger marker and immediately triggers the generation of its local data snapshot.

[0024] The above technical solution, which controls the generation time of each shard snapshot through the GTIDs snapshot (Global Transaction Identifier Snapshot) mechanism, has the following advantages: First, the snapshot generation triggered by each slave node can be completed asynchronously and concurrently. It does not require each standby machine to complete at the same time in real physical time. Instead, consistency is achieved through the same point in logical timing, which greatly reduces the complexity of distributed coordination. Second, there is no need for additional interaction between standby machines to negotiate the snapshot generation time. Each machine can independently identify the trigger point in log replay. Third, there is already a globally active residual transaction list on the standby machine. This list is stored in the snapshot transaction table, which facilitates the rollback of residual transactions when the image library starts up.

[0025] In summary, this application has the following beneficial effects: First, it consumes less disk space. The snapshot volume generated by this application shares the same full data as the original volume. When a write operation occurs in the mirror library, only the data block involved in the write operation is added. The increase in disk space is strictly linearly related to the scale of the write data in the mirror library. Since the full database data occupies the majority of disk storage, while the modification volume of the mirror library is usually small, this advantage is very obvious and can significantly save storage costs.

[0026] Secondly, the generation speed is fast. Generating a snapshot volume only requires copying the metadata of the logical volume, with a time overhead in the millisecond range. Compared to copying the entire database, this reduces the time by several orders of magnitude. Furthermore, because dirty data in memory is written back to disk before the snapshot volume is generated, there is no need to replay the write-ahead log when the mirror database starts, resulting in faster mirror database mounting and startup times. Overall, mirror database generation can be achieved in minutes.

[0027] Third, global consistency. By treating the request to generate a snapshot volume as an update transaction to the snapshot transaction table, this global transaction controls multiple shards to complete snapshot generation in the same logical time sequence, ensuring that snapshot volumes are generated logically at the same moment. Before generating the snapshot volume, dirty data in memory is flushed to disk to ensure that committed data is not lost. When writing back dirty data, the table is set to write-prohibited to avoid interference from other ongoing transactions during snapshot generation. After the mirror database starts, uncommitted sub-transactions are rolled back through the snapshot transaction table, eliminating residual sub-transactions on the distributed database, thereby achieving true global consistency.

[0028] Fourth, there is no impact on the production database. The production database experiences no blocking during mirror database creation and no performance impact during mirror database usage. The snapshot volume of the mirror database is created on the standby node, separate from the primary node of the production database. There are no high-load operations on the primary node during creation; the table locking action for flushing dirty data is performed on the standby node, having no impact on the primary database. The mirror database is co-located with the slave node but separate from the primary node; read and write operations on the mirror database have no impact on the primary node.

[0029] Figure 3 This is a flowchart of an optional distributed database mirror library implementation method according to an embodiment of this application, such as... Figure 3 As shown, the method may include the following steps: Step S302: The standby node synchronizes the transaction log from the corresponding master node and performs log replay to generate a data snapshot volume.

[0030] Step S304: Mount the data snapshot volume through the mount node corresponding to the standby node to provide the mirror library service.

[0031] By limiting the snapshot generation operation to the standby node and restricting it to simply synchronizing transaction logs from the master node and performing log replay to obtain a data copy consistent with the master node, the snapshot operation does not need to be performed on the master node. Therefore, the master node does not need to pause any requests due to the snapshot operation, thus resolving the technical problem of data synchronization causing business blockage in related technologies. This avoids pausing all requests and causing severe blockage to production services when performing snapshot operations directly on the production master node.

[0032] As an optional embodiment, the technical solution of this application is further described in detail below with reference to the embodiments: Please refer to Figure 4 and Figure 2 ,in Figure 4 The process of controlling the generation time of each slice snapshot through gtids snapshot is shown. Figure 2 The overall solution architecture diagram is shown. (For example...) Figure 4 As shown, the production database side includes a CN coordinating node, a GTM global transaction management node (primary), and each DN primary node; the standby database side includes each DN standby node; and the image database side includes each DN mounting node. When the CN initiates a distributed snapshot creation request, it obtains the global active transaction list from GTM and requests a GTID. Then, it simultaneously writes snapshot point data to the snapshot transaction table for each shard. This operation is committed as a distributed transaction. Each DN primary node generates the log of this transaction and replicates it to the corresponding DN standby node. Each DN standby node triggers data snapshot generation when replaying the transaction log, generating a data snapshot on local storage. In this way, each shard standby node generates a snapshot when replaying to the same global transaction log point, thereby achieving simultaneous alignment of snapshots for each shard in logical timing. Figure 2 The solution was further demonstrated from the perspective of the overall architecture, including the CN, GTM master, and DN master nodes on the production database side, as well as the DN standby nodes and DN mount nodes deployed on each server on the image database side. It clearly demonstrates the mechanism by which gtids snapshot controls the generation time of each shard snapshot in a distributed environment.

[0033] The complete process of gtids snapshot involves the following steps to initiate global snapshot generation: The first step is that the coordinating node (CN) receives a request to create a distributed snapshot. In practice, this can be initiated by executing an SQL command such as "ALTER DATABASE SNAPSHOT;", which is sent by the application or operations personnel through a connection to the coordinating node (CN). Upon receiving the command, the CN parses its content and identifies it as a distributed snapshot creation operation.

[0034] The second step involves the CN retrieving the list of currently active global transactions from the GTM and requesting a GTID value for its own transaction. The GTM, or Global Transaction Management Node, is responsible for maintaining the status information of global transactions, including which transactions are active (i.e., started but not yet committed). The CN initiates a query request to the GTM to obtain a list of GTIDs for all currently active transactions, and simultaneously requests a new GTID for the transaction itself created for this snapshot. This GTID will serve as a globally unique identifier for this snapshot, used to associate the same snapshot across all shards.

[0035] Third, CN simultaneously writes snapshot point data to each shard and commits it within the same distributed transaction. Specifically, CN inserts its current GTID value and the global active transaction list as a single record into the snapshot transaction table (SNAPSHOT_TRANS) of each shard. The snapshot transaction table is a replicated table, meaning that the data in this table is completely consistent across each shard node. CN ensures through the distributed transaction mechanism that this insertion operation either succeeds completely or fails completely on all shards, guaranteeing that the snapshot trigger records on each shard are consistent.

[0036] The fourth step involves each shard's master node generating a log for the transaction, which is then replicated to the corresponding slave node. Each data shard in the distributed database employs a master-slave replication architecture. All data modification operations on the master node generate binary logs (binlogs), which are synchronized to the slave node in real time via a log replication mechanism. The slave node continuously receives the log streams sent by the master node and replays these logs sequentially to maintain data consistency with the master node.

[0037] Fifth, when each slave node receives the log, it triggers a data snapshot generation during the replay process. During log replay, the slave node executes each transaction committed by the master node. When replaying a transaction that inserts a record into the snapshot transaction table, the slave node recognizes this as a snapshot-triggered transaction, pauses replaying subsequent logs, and begins the snapshot generation process.

[0038] The sixth step involves replaying all archived logs prior to the transaction on the standby node. Before triggering a snapshot, the standby node ensures that it has replayed all logs related to the triggering transaction and all preceding logs, ensuring that the standby node's data state is completely consistent with the master node's state when the transaction was committed. This step ensures the integrity of the snapshot data.

[0039] The seventh step is to flush dirty data from memory to disk from the nodes. To ensure that the data files on disk contain data from all committed transactions, dirty pages (i.e., data pages that have been modified but not yet written to disk) in the database buffer pool need to be flushed to disk. In practice, this can be achieved by executing an operation like "FLUSH TABLES WITH READ LOCK;", which flushes all table data to disk and applies read locks to the tables to prevent new write operations during the flushing process.

[0040] Step 8: Execute an LVM snapshot on the node to generate a copy-on-write data snapshot. LVM (Logical Volume Manager) provides snapshot functionality, allowing you to create a snapshot volume based on an existing logical volume. In practice, you can use the `lvcreate` command to create a snapshot, for example, `lvcreate -L 10G -s -n mysql_snap / dev / vg_data / lv_mysql`, where the `-s` parameter indicates that a snapshot is being created, `-L` specifies the size of the snapshot volume, and `-n` specifies the name of the snapshot volume. Snapshot creation uses a copy-on-write (COW) mechanism, copying only the metadata, making it very fast.

[0041] Step nine involves mounting the data snapshot to the corresponding path from the node. After the snapshot volume is created, it needs to be mounted to a directory in the file system so that the mounting node can access the data in the snapshot volume. In practice, the `mount` command can be used for mounting, for example, `mount / dev / vg_data / mysql_snap / mnt / snap`, and it can be set to read-write mode, allowing the mirror repository to perform write operations on the snapshot volume.

[0042] Step 10: Restore data read and write operations from the node. After the snapshot is generated and mounted, release the read locks on the table and restore normal database read and write operations. In practice, a command like "UNLOCK TABLES;" can be executed to release the read locks.

[0043] Step 11: The slave node records the mount path of the data snapshot in the snapshot volume path table of the production database. The recorded content includes the GTID value of the snapshot, the slave node identifier (such as the slave node's IP address and port number), and the snapshot generation path. The snapshot volume path table (SNAPSHOT_DIR) is used to store information about the snapshot volumes of each node for subsequent lookup during mirror database mounting and startup.

[0044] Step 12: The slave node continues to catch up with the master node. After the snapshot is generated, the slave node resumes normal log replay and continues to synchronize the transaction logs subsequently generated by the master node, maintaining data synchronization with the master node.

[0045] In some optional embodiments of this application, the snapshot transaction table is a replication table where the data is consistent across all data shard nodes. Its fields include: the global transaction number that generates the snapshot, used to associate the same snapshot across multiple data shards; and a global active transaction list, used to record the distributed transaction numbers that have not yet been committed at the time of snapshot generation, so that the corresponding residual sub-transactions can be rolled back after the mirror library starts.

[0046] In practical implementation, the Snapshot Transaction Table (SNAPSHOT_TRANS) is a system table in the distributed database. It employs a replication mechanism to ensure that each data shard node has an identical copy of the data. This table primarily contains two core fields: the first field is the transaction number that generated the snapshot. Its data type is typically a global transaction identifier, such as a large integer or a string (GTID). This field represents the global transaction number of the distributed transaction that triggered the snapshot generation, used to associate the same snapshot across multiple data shard nodes as a unique identifier for the global snapshot. The second field is the active transaction list, recording the numbers of all distributed transactions that have not yet been committed at the time of snapshot generation (i.e., the time the transaction commit is triggered). This list is obtained by the coordinating node (CN) by querying the GTM (Global Transaction Manager). Each element in the list is the GTID of an active transaction. Because transactions are processed in parallel across shards in the distributed database, at the time of snapshot generation, some cross-shard distributed transactions may have already started execution on some shards but have not yet been globally committed. These sub-transactions on different shards are called residual sub-transactions and need to be rolled back after the mirror database starts to ensure global consistency.

[0047] By adopting the above technical solution, and by designing a snapshot transaction table and utilizing a replication table mechanism, two important functions are achieved: First, the snapshots of each shard are associated with the same global snapshot through the global transaction number, providing an identification basis for the unified management of multi-node snapshots; second, by storing a list of globally active transactions in the snapshot transaction table, a basis is provided for rolling back residual sub-transactions when the mirror database starts up. This is the key data foundation for ensuring the global consistency of the distributed database mirror database.

[0048] In some optional implementations of this application, after each backup node completes the snapshot generation, it records the global transaction identifier, node identifier, and snapshot volume mount path of the snapshot in the snapshot volume path table. The snapshot volume path table is used to query the snapshot volume information corresponding to each node when the image library is mounted and started.

[0049] In the implementation process, the Snapshot Volume Path Table (SNAPSHOT_DIR) is a system table that records snapshot volume information for each node. This table mainly contains three fields: the first field is the snapshot time point, i.e., the Global Transaction ID (GTID) that generated the snapshot volume, used to identify the globally consistent time point corresponding to the snapshot; the second field is the DB node identifier, used to identify which database node generated the snapshot, usually a combination of the node's IP address and port number, such as "10.10.10.1:3306"; the third field is the snapshot generation path, i.e., the mount path of the snapshot volume on that node, such as " / mnt / snap", used to record the specific location of the snapshot volume in the file system. After each standby node completes snapshot generation and mounting, it inserts the above information into the Snapshot Volume Path Table. Since each node generates snapshots independently, each node can asynchronously and concurrently insert records into this table. When the mirror repository needs to be started, the coordinating node can query this table to obtain the snapshot volume path information of all nodes corresponding to a specified GTID, thereby guiding each mounting node to mount the corresponding snapshot volume.

[0050] By employing the above technical solution, the snapshot volume path table records the generation location of snapshots for each node and the corresponding global transaction number, providing an information foundation for the rapid mounting and startup of the image repository. The image repository coordinating node only needs to query this table based on the target GTID to obtain the snapshot volume location information of all shards, and then schedule each mounting node to complete the startup, thus decoupling the image repository generation from the mounting process.

[0051] In some optional implementations of this application, a mirror repository mounting and startup and residual transaction rollback mechanism is adopted. The mirror repository cluster also includes a mirror repository coordinating node, which is used to schedule each mounting node to roll back the residual sub-transactions corresponding to the globally active transaction list recorded in the snapshot transaction table when the mirror repository is mounted and started, so that the mirror repository reaches a globally consistent state.

[0052] In some optional implementations of this application, the mirror library mounting and startup steps are as follows: receiving a mount command, the mount command carrying a target global transaction identifier; obtaining the snapshot volume mount path corresponding to each node from the snapshot volume path table according to the target global transaction identifier; switching the data path of each mount node to the corresponding snapshot volume mount path and restarting; after each mount node starts, rolling back the residual sub-transactions corresponding to the global active transaction list recorded in the snapshot transaction table, so that the mirror library reaches a globally consistent state.

[0053] In practice, after the production backup machine creates a snapshot volume for each DN node, the image library mounting process can be initiated. The specific steps for mounting and starting the image library are as follows: The first step is to connect to the image repository CN and execute the mount command. Operations personnel or applications connect to the image repository's coordinating node CN and execute the mount command, for example, "ALTER DATABASE LOAD SNAPSHOT ON GTID = '0323424234234234';", where the GTID parameter specifies the global transaction identifier corresponding to the snapshot to be loaded, i.e., which snapshot from which point in time is selected to start the image repository.

[0054] The second step is for the mirror repository CN to query the snapshot volume information from the production repository CN. The mirror repository CN retrieves the snapshot volume information for each node corresponding to the specified GTID by querying the snapshot volume path table (SNAPSHOT_DIR), including the node identifier and snapshot mount path.

[0055] Third, the mirror repository (CN) sends a request to each mount point (DN), requesting it to switch its data path to the corresponding mount volume path for startup. Based on the retrieved snapshot volume information, the CN sends startup commands to each mount node, which include the mount path of the corresponding data snapshot volume for that node.

[0056] Fourth, the mount point DN switches its datadir path to the corresponding mount volume path and then restarts. After receiving the instruction, each mount node modifies its own data directory configuration, pointing the data directory to the mount path of the snapshot volume, and then restarts the database instance based on the data under that path.

[0057] Fifth, after the DN starts, it returns the mounting results to the CN. After each mounted node has started, it reports its startup status to the image repository CN. The CN collects the startup results of all nodes and confirms that all shards have started successfully.

[0058] Step 6: CN continues to schedule DNs, instructing each DN to roll back the list of active transactions in the snapshot transaction table. After all nodes have started, CN reads the global list of active transactions corresponding to the snapshot from the snapshot transaction table, and then schedules each mounted node to perform rollback operations on these active transactions.

[0059] Step 7: After the DN rollback is complete, it returns the results to the CN. Each mounted node rolls back the sub-transactions on its shard that belong to the active transaction list, according to the CN's instructions. After the rollback is complete, each node returns the rollback results to the CN. At this point, all remaining distributed sub-transactions on the mirror repository have been cleaned up, and the mirror repository is in a state that is completely consistent with the production repository at the time of the snapshot and is globally consistent.

[0060] Step 8: The CN completes the entire mounting and startup command, and applications can begin connecting to the CN to query the image repository. After the CN confirms that all nodes have completed the startup and rollback operations, it returns a successful mounting response to the client. At this point, the image repository can provide services normally, and applications can connect to the image repository CN to perform operations such as data querying, report generation, and application verification.

[0061] Please refer to Figure 4 and Figure 5 ,in Figure 5 A schematic diagram illustrating the principle of the COW (Copy-on-Write) mechanism is shown. Figure 4 This illustrates the removal of globally active transactions after the mirror repository starts (i.e., distributed rollback to a globally consistent state). Figure 5 As shown, the Copy-on-Write (COW) mechanism consists of three phases: The first phase occurs when a snapshot is created using the `lvcreate -s` command. At this stage, only metadata is copied, and the COW table is empty. The second phase occurs when the snapshot volume is read. Reading snapshot data first queries the COW table. If the data block has no record in the COW table, it means the data block has not been modified, and the data is read from the original volume. The third phase occurs when a write to the original volume triggers COW. When the original volume needs to write a data block (e.g., writing a new value Y to block B), LVM first copies the old value (X) of the data block to the snapshot volume, then updates the COW table, recording the mapping of block B to the corresponding position in the snapshot volume, and finally writes the new value Y to the data block in the original volume. This copy-on-write mechanism enables rapid snapshot generation and efficient space utilization.

[0062] like Figure 4 As shown, the production database side includes the CN, GTM primary, and each DN primary node; the standby database side includes each DN standby node, with each shard's snapshot point location marked using gtids snapshots; the mirror database side includes each DN mount node. After the mirror database starts, globally active transactions need to be removed, that is, all sub-transactions corresponding to the active transaction list recorded in the snapshot transaction table need to be rolled back, so that the mirror database can reach a globally consistent state. Figure 4 The document clearly demonstrates the data flow from the production database to the standby database and then to the mirror database, as well as the process of achieving global consistency by rolling back residual transactions after the mirror database is started.

[0063] In some optional embodiments of this application, the copy-on-write mechanism is described in detail. The working principle of the copy-on-write mechanism is as follows: when creating a snapshot volume, only the metadata of the original volume is copied, and the mapping relationship of the modified data blocks is recorded through the COW table; when reading snapshot data, the COW table is queried first. If there is no record for the data block, it is read from the original volume; if there is a record, it is read from the snapshot volume; when writing data to the original volume, the old value of the data block to be modified is copied to the snapshot volume, the mapping relationship in the COW table is updated, and then the new value is written to the data block of the original volume.

[0064] In practical implementation, Copy-on-Write (COW) is a storage snapshot technology. Its core idea is not to copy all data immediately when creating a snapshot, but rather to copy the modified data blocks only when the data is modified. LVM (Logical Volume Manager) is a commonly used logical volume management tool in Linux systems, and it has built-in support for COW snapshot functionality. Specifically, the working principle of the COW mechanism is as follows: First, when creating a snapshot volume, the system does not copy the actual data from the original volume. Instead, it only copies the metadata information, such as the block allocation table. Simultaneously, it creates a Copy-on-Write (COW) table to record which data blocks in the original volume have been modified and the location of the modified data in the snapshot volume. Initially, the COW table is empty, indicating that all data blocks have not yet been modified, and the data in the snapshot volume is completely identical to that of the original volume. Because only metadata is copied, snapshot creation is very fast, typically completed in milliseconds, regardless of the size of the original volume's data.

[0065] Secondly, when reading data from a snapshot volume, the system first queries the COW table to check if the data block to be read has a record in the COW table. If the COW table does not contain a record for the data block, it means that the data block has never been modified since the snapshot was created, and the data in the snapshot volume is exactly the same as in the original volume. Therefore, the system directly reads the data block from the original volume and returns it. If the COW table contains a record for the data block, it means that the data block has been modified, and the old data has been copied to a specific location in the snapshot volume. In this case, the system reads the old value of the data block from the corresponding location in the snapshot volume and returns it.

[0066] Secondly, the Copy-on-Write (COW) mechanism is triggered when data is written to the original volume, specifically when a data block in the original volume needs to be modified. The process is as follows: The system first checks if the data block already has a record in the COW table. If there is no record, it means this is the first time the data block has been modified since the snapshot was created. The system needs to copy the current value (old value) of the data block from the original volume to the snapshot volume and add a record to the COW table, recording the data block's number and its storage location in the snapshot volume. Only after the copying and COW table update are completed does the system write the new data to the corresponding data block in the original volume. If the data block already has a record in the COW table, it means it has been modified before, and the old data is already stored in the snapshot volume. In this case, the new data can be written directly to the original volume without triggering the COW operation again.

[0067] Through this mechanism, the snapshot volume shares most of the unmodified data blocks with the original volume; only modified data blocks occupy additional storage space in the snapshot volume. This results in minimal space overhead for the snapshot volume, especially when data modifications are infrequent, as the storage space occupied by the snapshot volume is far less than the total capacity of the original volume. Furthermore, since snapshot creation only requires copying metadata, the generation speed is extremely fast, making it ideal for applications that require rapid and repeated generation of image libraries.

[0068] In the scheme of this application, the data volume of the standby node serves as the original volume, and the snapshot volume used by the mounting node is generated based on this original volume through a Copy-on-Write (COW) mechanism. The standby node continuously receives and replays the transaction logs of the primary node, which is equivalent to continuously writing to the original volume. These writes trigger COW, copying the old values ​​of the modified data blocks to the snapshot volume. The read and write operations of the mounting node on the snapshot volume are implemented through the mapping of the COW table. In this way, the standby node and the mounting node can run simultaneously, their data modifications do not interfere with each other, and they share most of the unmodified data, greatly saving storage space.

[0069] To better understand the technical advantages of this application, the following describes existing solutions based on Distributed Database Cluster Replication (DRSP) technology. Please refer to... Figure 6 , Figure 6 This illustrates a mirror repository generation scheme based on Cluster Replication SP (DRSP) technology. For example... Figure 6 As shown, this solution comprises three parts: a production database, a standby machine, and a mirror database. The basic process is as follows: ① Replication stops at the application time point in the mirror database (e.g., 00:00 AM); ② A backup of the local data directory in the mirror database is created for later use, for example, copying / gdbdata_A to / gdbdata_B; ③ The mirror database now matches the production database data and switches to read-write mode, writing the application data to / gdbdata_B; ④ The standby machine resumes the replication relationship, continuing to track data from 00:00 AM to the present and writing it to / gdbdata_A.

[0070] The main problem with this approach is its inability to quickly regenerate the mirror repository. Because if the mirror repository is modified and a new one needs to be generated, the data must be rolled back to its initial state before being caught up with the latest data in the source repository. If the amount of data to be rolled back or the amount of data to be caught up is large (typically more than one day's worth), the time to catch up with the latest data the second time will be very long, failing to meet the requirement of minute-level rapid generation. In contrast, the approach proposed in this application utilizes copy-on-write technology to generate snapshot volumes. Each mirror repository generation requires only millisecond-level snapshot creation time, and can be repeatedly and rapidly generated, significantly improving the efficiency of mirror repository generation.

[0071] The technical solution of this application can be widely applied to various scenarios that require the rapid construction of independent database environments based on production data. Some typical application scenarios are listed below: Scenario 1: Off-site query and report generation. In enterprise business systems, a large number of data queries and statistical report operations consume significant database resources, impacting the performance of online transaction systems. This solution allows for the rapid generation of a mirror database, migrating query and reporting operations to this database for execution. This ensures the real-time nature and consistency of query data while completely eliminating the performance impact on the production database. Since the mirror database can be generated in minutes, it can be updated periodically, either hourly or daily, to meet query needs with varying timeliness requirements.

[0072] Scenario 2: Application Version Verification. Before upgrading or releasing a new version of an application system, verification is typically performed in a data environment consistent with the production environment to ensure that the new version can correctly process production data. Traditional methods require significant time to copy production data to the test environment. However, this solution can generate a mirror repository that is completely identical to the production repository within minutes, providing a powerful tool for application version verification. This significantly shortens the verification cycle and improves release efficiency. After verification, the snapshot volume can be deleted to free up space, and it can be regenerated for the next verification.

[0073] Scenario 3: System Implementation Drills. For critical business systems such as those in finance and telecommunications, regular disaster recovery drills are essential. This solution can quickly generate a mirror database to simulate the database environment after a disaster recovery, used to practice disaster recovery processes and verify the effectiveness of recovery plans. Because the generation is fast and has no impact on production, drills can be performed frequently, improving the system's disaster recovery capabilities.

[0074] Scenario 4: Data Anonymization and Development / Testing. Development and testing teams need to use real data for development and testing, but directly using production data poses data security risks. This solution generates a mirror repository, on which data anonymization can be performed, replacing sensitive data with simulated data. The anonymized mirror repository is then provided to the development and testing teams. This ensures both data authenticity and consistency while protecting the security of production data.

[0075] Scenario 5: Data Analysis and Data Mining. Big data analysis and data mining tasks typically require scanning and computing the entire dataset, which significantly impacts the performance of the production database. Using the mirror database generated by this solution, analysis tasks can be migrated to the mirror database for execution, avoiding interference with the production system. Furthermore, because the mirror database can be generated repeatedly and quickly, data snapshots at different points in time can be created according to different analytical needs, supporting historical data backtracking analysis.

[0076] The above application scenarios are merely illustrative examples, and those skilled in the art can apply this solution to other similar scenarios according to actual needs.

[0077] As mentioned above, the technical solution of this application has significant technical effects in four aspects: disk usage, generation speed, global consistency, and impact on the production database. A more detailed analysis follows.

[0078] Regarding the low disk usage, the snapshot volumes generated in this application share the same full data as the original volumes. Modified data blocks are only copied when write operations occur in the mirror repository or standby node. Therefore, the increase in disk space is strictly linearly related to the amount of data involved in the write operation. In typical mirror repository application scenarios, such as off-site querying, report generation, and application verification, write operations on the mirror repository are usually infrequent, and most data is read-only accessed. Therefore, the additional storage space occupied by the snapshot volumes is very limited. Compared to full replication, this solution can save more than 90% of storage space, significantly reducing storage costs. In addition, the space efficiency of the COW mechanism is also reflected in the ability to delete unnecessary snapshot volumes at any time to free up space, supporting flexible space management.

[0079] Regarding the technical benefits of rapid generation, this solution's speed is reflected in two aspects: First, snapshot generation is fast. Creating a snapshot volume only requires copying the metadata of the original LVM volume, with a time overhead in the millisecond range. Regardless of whether the data volume is in the GB or TB range, the snapshot creation time remains almost unchanged. Compared to the time it takes for full data copying to be several hours or even days, this solution reduces the mirror repository generation time by several orders of magnitude. Second, the mirror repository starts up quickly. Because a dirty page flushing operation has been performed before snapshot generation, writing all committed data in memory to disk, the mirror repository does not need to replay the redo log when starting from the snapshot volume. It can directly open the database and provide services, further shortening the mirror repository startup time. In summary, the entire mirror repository generation process can be completed in minutes, fully meeting the requirements for rapid generation.

[0080] Regarding the technical effectiveness of global consistency, this solution ensures the global consistency of the mirror database through a multi-layered guarantee mechanism. The first layer is logical time-series alignment. Through the gtids snapshot mechanism, backup nodes on each shard generate snapshots when replaying to the same global transaction log point, ensuring logical time consistency across shard snapshots. The second layer is dirty page flushing. Before generating a snapshot, a FLUSH TABLES WITH READ LOCK operation is performed to flush all dirty data in memory to disk. Read locks prevent new writes during the flushing process, ensuring that the data files on disk contain data from all committed transactions. The third layer is residual transaction rollback. Since there may be active transactions across shards in the distributed database, these transactions may not have been committed when the snapshot is generated. Their sub-transactions on each shard may be in a partially committed state. After the mirror database starts, these active transactions are rolled back, eliminating residual uncommitted sub-transactions and ensuring that the data in the mirror database is all globally committed data. Through these three mechanisms, this solution achieves a truly distributed, globally consistent mirror database.

[0081] Regarding the technical effectiveness of having no impact on the production database, this solution ensures zero impact on the production database from multiple dimensions. First, the deployment is completely separate; the mirror database and the production database do not share servers. All snapshot-related computations and I / O operations are performed on the standby machine, without consuming the CPU, memory, and I / O resources of the production database's primary node. Second, the snapshot triggering method is non-intrusive. Snapshot generation is triggered by log replay on the standby node. The production database's primary node only needs to handle transactions and synchronize logs normally, without performing any special snapshot operations or pausing business requests. Third, table locking operations are performed on the standby machine. The table locking action (FLUSH TABLES WITH READ LOCK) during dirty page flushing is performed on the standby node, having no impact on the production database's primary node. Finally, there is no interference during the use of the mirror database. All read and write operations on the mirror database are performed on the mounted node, which runs on an independent snapshot volume, isolated from the original volume of the standby node through a COW mechanism. Any operation on the mirror database will not affect the data and performance of the production database.

[0082] In summary, this application, through an innovative deployment architecture that separates the mirror repository from the production repository and co-locates standby nodes and mounting nodes, combined with the gtids snapshot triggering mechanism, COW copy-on-write snapshot technology, and residual transaction rollback mechanism, achieves rapid generation of a globally consistent mirror repository for a distributed database based on local storage. It has significant advantages such as low disk usage, fast generation speed, global consistency, and no impact on the production repository, effectively solving the technical problem that traditional solutions cannot quickly generate a globally consistent mirror repository in a distributed database local storage environment.

[0083] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0085] According to another aspect of the embodiments of this application, a mirror library implementation apparatus for implementing the above-described distributed database mirror library implementation method is also provided. The apparatus may include: The generation module is used to instruct the standby node to synchronize transaction logs from the corresponding master node and perform log replay to generate data snapshot volumes; The mounting module is used to instruct the mounting node corresponding to the backup node to mount the data snapshot volume, thereby providing a mirror library service.

[0086] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in a corresponding hardware environment, and can be implemented through software or hardware, wherein the hardware environment includes a network environment.

[0087] According to another aspect of the embodiments of this application, a server or terminal for implementing the above-described distributed database mirror library implementation method is also provided.

[0088] Figure 7 This is a structural block diagram of a terminal according to an embodiment of this application, such as... Figure 7 As shown, the terminal may include: one or more (only one is shown in the figure) processors 901, memory 903, and transmission devices 905, such as... Figure 7 As shown, the terminal may also include input / output devices 907.

[0089] The memory 903 can be used to store software programs and modules, such as the program instructions / modules corresponding to the distributed database mirror library implementation method and apparatus in this embodiment. The processor 901 executes various functional applications and data processing by running the software programs and modules stored in the memory 903, thereby implementing the aforementioned distributed database mirror library implementation method. The memory 903 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 903 may further include memory remotely located relative to the processor 901, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0090] The aforementioned transmission device 905 is used to receive or send data via a network, and can also be used for data transfer between a processor and memory. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 905 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 905 is a radio frequency (RF) module used for wireless communication with the Internet.

[0091] Specifically, memory 903 is used to store application programs.

[0092] The processor 901 can invoke the application program stored in the memory 903 via the transmission device 905 to perform the following steps: The standby node synchronizes the transaction log from the corresponding master node and performs log replay to generate a data snapshot volume; The data snapshot volume is mounted by the mounting node corresponding to the backup node, thereby providing a mirror library service.

[0093] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0094] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only. The terminal can be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal devices. Figure 7This does not limit the structure of the aforementioned electronic device. For example, the terminal may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 7 The different configurations shown.

[0095] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0096] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to execute program code for a distributed database mirror library implementation method.

[0097] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0098] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: The standby node synchronizes the transaction log from the corresponding master node and performs log replay to generate a data snapshot volume; The data snapshot volume is mounted by the mounting node corresponding to the backup node, thereby providing a mirror library service.

[0099] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0100] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0101] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0102] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0103] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A mirror library implementation system for a distributed database, characterized in that, include: The system comprises a production database cluster and a mirror database cluster. The production database cluster includes a coordination node, a global transaction management node, and multiple data shard nodes. Each data shard node includes a primary node and a backup node. The mirror database cluster includes multiple mount nodes. Each backup node corresponds one-to-one with a mount node and is deployed on the same physical device. The backup node is used to synchronize transaction logs from the corresponding primary node and perform log replay to generate data snapshot volumes. The mount nodes are used to mount data snapshot volumes and provide mirror database services.

2. The mirror library implementation system according to claim 1, characterized in that, The coordinating node is used to respond to snapshot creation requests, obtain a global transaction identifier and a global active transaction list from the global transaction management node, and write the global transaction identifier and the global active transaction list into the snapshot transaction table of each data shard node. The snapshot transaction table is a replication table shared by each data shard node. When each backup node identifies the log position corresponding to the write operation of the snapshot transaction table during log replay, it triggers a data snapshot generation operation.

3. The mirror library implementation system according to claim 2, characterized in that, The image repository cluster also includes an image repository coordinating node, which is used to schedule the residual sub-transactions corresponding to the globally active transaction list recorded in the snapshot transaction table of each mounting node when the image repository is mounted and started, so that the image repository can reach a globally consistent state.

4. A method for implementing a mirror repository for a distributed database, characterized in that, A mirror library implementation system for a distributed database according to any one of claims 1 to 3, wherein the mirror library implementation method includes: The standby node synchronizes the transaction log from the corresponding master node and performs log replay to generate a data snapshot volume; The data snapshot volume is mounted by the mounting node corresponding to the backup node, thereby providing a mirror library service.

5. The mirror library implementation method according to claim 4, characterized in that, The standby node synchronizes transaction logs from the corresponding master node and performs log replay, generating a data snapshot volume, including: In response to the snapshot creation request, the coordinating node requests the allocation of a global transaction identifier from the global transaction management node and obtains the current list of global active transactions; The coordinating node inserts the global transaction identifier and the global active transaction list as a record into the snapshot transaction table of each data shard node, and commits the insertion operation in the same distributed transaction; The master node of each data shard node generates a transaction log containing the insertion operation and copies the transaction log to the corresponding backup node; When the backup node identifies the log position corresponding to the insertion operation during log replay, it generates a data snapshot volume, wherein the log position corresponding to the insertion operation serves as a logical timing alignment marker for each backup node to generate snapshot volumes.

6. The mirror library implementation method according to claim 5, characterized in that, Generating a data snapshot volume includes: After completing the replay of all logs up to the specified log point, the dirty data pages in memory are flushed to the disk, and a data snapshot volume is generated through the copy-on-write mechanism of logical volume management, so that the data snapshot volume can be mounted to the specified path.

7. The mirror library implementation method according to claim 6, characterized in that, Data snapshot volumes are generated through the write-on-write replication mechanism of logical volume management, including: When creating the data snapshot volume, only the metadata of the original volume is copied; When the mounted node writes new data to the data snapshot volume, it copies the old value of the data block to be modified to the copy-on-write storage area of ​​the data snapshot volume, updates the mapping relationship in the copy-on-write table, and writes the new data to the corresponding new data block in the copy-on-write storage area.

8. The mirror library implementation method according to claim 5, characterized in that, Before providing the mirror repository service by mounting the data snapshot volume through the mounting node corresponding to the backup node, the mirror repository implementation method further includes: After the backup node completes the snapshot generation, it records the corresponding global transaction identifier, node identifier, and snapshot volume mount path into the snapshot volume path table. The snapshot volume path table is used to query the corresponding snapshot volume information when the image library is mounted and started.

9. The mirror library implementation method according to claim 5, characterized in that, Before providing the mirror repository service by mounting the data snapshot volume through the mounting node corresponding to the backup node, the mirror repository implementation method further includes: The corresponding snapshot volume mount path is obtained from the snapshot volume path table based on the global transaction identifier; Switch the data path of each mounted node to the corresponding snapshot volume mount path and restart; After each mounted node starts up, the residual sub-transactions corresponding to the global active transaction list are recorded by rolling back the snapshot transaction table, so that the mirror library reaches a globally consistent state.

10. The mirror library implementation method according to any one of claims 5 to 9, characterized in that, The snapshot transaction table is a replication table that ensures data consistency across all data shard nodes. The fields in the replication table include: the global transaction number that generated the snapshot, the same snapshot used to associate multiple data shards, a global list of active transactions, and the distributed transaction number used to record the uncommitted transactions at the time the snapshot was generated.