Rapid data backup and recovery method and device for TiDB distributed database based on CDM technology

Abstract

The embodiment of the invention discloses a rapid data backup and recovery method and device for a TiDB distributed database based on a CDM technology. The method comprises the following steps: creating a storage volume on a CDM backup platform, configuring a backup strategy and deploying a client; obtaining an operation request; the TiDB distributed database data is backed up by distributing a storage volume, scheduling a job task and monitoring the progress in real time; and selecting a backup data set, and recovering the backup data set to a specified target cluster by using a snapshot technology and a multi-node cooperation function. By means of the method, rapid and consistent recovery can be achieved, backup and recovery operation is remarkably simplified, efficiency is improved, and the method is particularly suitable for the strict disaster recovery time target requirement of the financial industry.

Description

Technical Field

[0001] The present invention relates to a data backup and recovery method, and more specifically to a TiDB distributed database fast data backup and recovery method and device based on CDM technology. Background Art

[0002] At present, the distributed nature of the TiDB database has prompted backup software vendors on the market to generally adopt TiDB's built-in BR (backup & restore) tool to perform backup tasks and store data in object storage media such as NFS or S3.

[0003] Although this is the mainstream backup and recovery solution, when performing database backup, it is usually necessary to mount the NFS file system on all hosts or prepare additional resources for object storage. This practice not only increases the invasiveness of the operating system, but also may bring additional complexity in management and maintenance. Moreover, TiDB backup will generate a large number of small files. In this case, the transmission efficiency is often low, resulting in a long backup process. In addition, a large number of small files will also increase storage requirements and management costs. Furthermore, when faced with large-scale data recovery tasks, the recovery time of this solution is often long, resulting in the RTO (Recovery Time Objective) may not meet business needs, especially when critical data needs to be restored quickly.

[0004] In summary, although the backup solutions currently on the market basically use TiDB's own tools, the challenges they face include invasive operating system requirements, inefficient backup transmission, and insufficient recovery efficiency.

[0005] Therefore, it is necessary to design a new method to achieve fast and consistent recovery, significantly simplify backup and recovery operations, and improve efficiency, which is particularly suitable for the strict disaster recovery time target requirements of the financial industry. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and provide a TiDB distributed database fast data backup and recovery method and device based on CDM technology.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: TiDB distributed database fast data backup and recovery method based on CDM technology, including:

[0008] Create storage volumes, configure backup policies, and deploy clients on the CDM backup platform;

[0009] Get operation request;

[0010] Back up TiDB distributed database data by allocating storage volumes, scheduling job tasks, and monitoring progress in real time;

[0011] Select a backup dataset and use snapshot technology and multi-node collaboration to restore the backup dataset to the specified target cluster.

[0012] The further technical solution is: creating storage volumes, configuring backup strategies and deploying clients on the CDM backup platform includes:

[0013] Create a storage volume on the CDM backup platform;

[0014] Configure the database backup policy on the CDM backup platform and select the original format for data backup;

[0015] Deploy the client on the control node of the TiDB distributed database and set it as the master node. Deploy the client on all TiKV nodes and set them as slave nodes.

[0016] A further technical solution is: the backup strategy includes setting a backup initiation time and an expiration time of a backup data set.

[0017] The further technical solution is: backing up TiDB distributed database data by allocating storage volumes, scheduling job tasks and real-time monitoring progress includes:

[0018] Initiate a backup job and allocate storage volumes from the storage server;

[0019] Analyze the topology of the TiDB database cluster, automatically identify TiKV nodes and control nodes, and send backup job tasks and parameters to the client of the control node. The client of the control node can obtain the consistency time point through pd-ctl, suspend GC and scheduling, and feedback the job status.

[0020] When the control node job is successfully executed, the backup job task is sent to the clients of the TiKV nodes of all TiDB databases, so that the clients of the TiKV nodes can take a snapshot of the file system through the CBT component of Cloud Trust and copy the snapshot data to the storage volume;

[0021] When all TiKV nodes complete the snapshot, the client of the scheduling control node re-enables the GC and scheduling functions.

[0022] Its further technical solution is: when the control node job is successfully executed, the backup job task is sent to the client of the TiKV node of all TiDB databases, so that the client of the TiKV node takes a snapshot of the file system through the CBT component of Yunxinda, and copies the snapshot data to the storage volume, including:

[0023] When the control node job is successfully executed, the backup job task is sent to the clients of the TiKV nodes of all TiDB databases, so that the TiKV node clients can obtain the directory of each TiKV data node and analyze the corresponding file system, use Yunxinda's CBT component to snapshot the file system, and copy the snapshot data to the storage volume.

[0024] Its further technical solution is: when the operation request is for TiDB distributed database data recovery, a backup data set is selected, and the backup data set is restored to a specified target cluster using snapshot technology and multi-node collaboration functions, including:

[0025] Select the data set to be restored, specify the target cluster, and start the restore job;

[0026] The job scheduling component automatically analyzes whether the target cluster meets the recovery conditions, parses the control node and TiKV node of the target cluster, and confirms whether the corresponding client has been installed on the control node and TiKV node;

[0027] When the target cluster meets the recovery conditions and the corresponding client has been installed on the control node and TiKV node, take a snapshot of the data set to be restored;

[0028] Send a job to the client of the control node so that the client of the control node can clear the target cluster and start the empty PD node, and feedback the status after success;

[0029] Mount the data in the restored storage volume to all TiKV nodes, connect the mounted data directory to the data directory of the TiKV node through a soft link, and modify the directory permissions;

[0030] When the clients of all TiKV nodes successfully mount the file system, the scheduling component sends the startup job of restoring the target cluster to the client of the control node, so that the client of the control node executes the startup job of restoring the target cluster until the client of the control node reports the successful recovery information.

[0031] Its further technical solution is: when the clients of all TiKV nodes successfully mount the file system, the scheduling component sends the startup job of restoring the target cluster to the client of the control node, so that the client of the control node executes the startup job of restoring the target cluster until the client of the control node reports the information of successful recovery, including:

[0032] When the clients of all TiKV nodes successfully mount the file system, the scheduling component sends a startup job to restore the target cluster to the client of the control node, so that the client of the control node switches the PD node to recovery mode, starts all TiKV services, stops the cluster scheduling function, uses the BR tool to restore the target cluster to the consistency time point at the time of backup, exits recovery mode, and restarts the TiDB database cluster.

[0033] The present invention also provides a TiDB distributed database fast data backup and recovery device based on CDM technology, including:

[0034] The backup preparation unit is used to create storage volumes, configure backup policies, and deploy clients on the CDM backup platform.

[0035] A request acquisition unit, used for acquiring an operation request;

[0036] The backup unit is used to back up TiDB distributed database data by allocating storage volumes, scheduling job tasks, and monitoring progress in real time;

[0037] The recovery unit is used to select a backup data set and restore the backup data set to a specified target cluster using snapshot technology and multi-node collaboration functions.

[0038] The present invention further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.

[0039] The present invention also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0040] Compared with the prior art, the present invention has the following advantages: the present invention creates storage volumes and configures backup policies through the CDM backup platform. After the client is deployed, the operation request is a backup request. The system allocates storage volumes and schedules tasks, and monitors the backup progress in real time. For non-backup requests but involving TiDB data recovery, the platform uses snapshot technology and multi-node collaboration functions to select and restore backup data sets to the specified target cluster, achieving fast and consistent recovery, significantly simplifying backup and recovery operations, and improving efficiency. It is particularly suitable for the strict disaster recovery time target requirements of the financial industry.

[0041] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0043] Figure 1 A schematic diagram of an application scenario of a TiDB distributed database fast data backup and recovery method based on CDM technology provided in an embodiment of the present invention;

[0044] Figure 2 A flowchart of a TiDB distributed database fast data backup and recovery method based on CDM technology provided in an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a sub-process of a TiDB distributed database fast data backup and recovery method based on CDM technology provided in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a sub-process of a TiDB distributed database fast data backup and recovery method based on CDM technology provided in an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of a sub-process of a TiDB distributed database fast data backup and recovery method based on CDM technology provided in an embodiment of the present invention;

[0048] Figure 6 A schematic block diagram of a TiDB distributed database fast data backup and recovery device based on CDM technology provided in an embodiment of the present invention;

[0049] Figure 7 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0052] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0053] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0054] See also Figure 1 and Figure 2 , Figure 1 Schematic diagram of application scenarios of the TiDB distributed database fast data backup and recovery method based on CDM technology provided in an embodiment of the present invention. Figure 2 A schematic flow chart of a TiDB distributed database fast data backup and recovery method based on CDM technology provided in an embodiment of the present invention. The TiDB distributed database fast data backup and recovery method based on CDM technology is applied to a CDM (Cloud Data Management) server. The CDM server is presented as a CDM backup platform. The CDM server interacts with the control node and TiKV node of the TiDB database for data. Specifically, the control node of the TiDB database and the corresponding client of the TiKV node are set to interact with data. The CDM backup platform provides powerful storage management and job scheduling capabilities. As the master node, the control node is responsible for managing the scheduling functions such as the start and stop, garbage collection (GC), and consistency point management of the TiDB database. The backup client is deployed on all TiKV nodes and is responsible for snapshotting and data transmission of the data disk during the backup process. Relying on Yunxinda's CBT technology, the ability to quickly recover and ensure database consistency points is achieved by snapshotting and copying the data node disks of the TiDB distributed database. It greatly simplifies the complexity of backup and recovery operations, making them easier to manage and execute; snapshots and data copies achieved through CBT technology significantly improve the efficiency of backup and recovery and shorten the recovery time (RTO); it supports database recovery to a specified consistency point in time to ensure data integrity and consistency in disaster recovery scenarios; it is particularly suitable for the strict requirements of the financial industry for disaster recovery time objectives (RTO), and can quickly respond to and restore key business systems. It not only provides powerful backup and recovery functions, but also effectively reduces the data risks faced by financial institutions and improves business continuity.

[0055] Figure 21 is a flow chart of a TiDB distributed database fast data backup and recovery method based on CDM technology provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S140.

[0056] S110. Create storage volumes on the CDM backup platform, configure backup policies, and deploy clients.

[0057] In one embodiment, see Figure 3 , the above-mentioned step S110 may include steps S111 to S113.

[0058] S111. Create a storage volume on the CDM backup platform.

[0059] In this embodiment, a storage volume for storing backup data is allocated on the CDM machine backup platform.

[0060] Isolate backup data from other data to ensure that backup data is more reliable and secure in storage and management. Managing backup data through separate storage volumes can simplify data management and optimize the efficiency of backup and recovery. Dedicated storage volumes can optimize the performance of the backup process while ensuring high availability and security of backup data.

[0061] S112. Configure the database backup strategy on the CDM backup platform, and select the data backup format as the original format.

[0062] In this embodiment, the backup strategy includes setting the backup initiation time and the expiration time of the backup data set. The data backup format is selected as the original format, that is, a backup of the current state of the complete copy of the database.

[0063] By backing up regularly, you can effectively protect the database from the risk of data loss or damage; backing up data in the original format can provide faster and more accurate database recovery and shorten the system recovery time (RTO).

[0064] S113. Deploy the client on the control node of the TiDB distributed database and set it as the master node. Deploy the client on all TiKV nodes and set them as slave nodes.

[0065] In this embodiment, the CDM client is installed on the TiDB control node and configured as the master node, which is responsible for initiating backup operations; the CDM client is installed on all TiKV nodes (TiKV is the storage node of TiDB) and configured as slave nodes for receiving backup commands and data transmission.

[0066] By deploying CDM clients on control nodes and storage nodes, complete backup management of the TiDB distributed architecture is achieved; the master node is responsible for coordinating backup operations, while the slave nodes are responsible for actual data transmission and backup storage, improving the efficiency and reliability of backup operations.

[0067] In summary, these steps and operations are aimed at establishing a comprehensive backup and recovery strategy to ensure data security, availability, and durability of the TiDB distributed database while optimizing the backup process and resource utilization.

[0068] S120, obtaining an operation request;

[0069] S130. Back up TiDB distributed database data by allocating storage volumes, scheduling job tasks, and monitoring progress in real time.

[0070] In this embodiment, when the user starts a backup task on the CDM backup platform, the CDM backup platform will automatically execute the following process: First, it will automatically divide the storage volume to ensure that there is enough space for the backup data. Then, the backup platform will schedule the clients deployed on the control nodes and TiKV nodes of each TiDB distributed database to complete the actual backup operation. These clients are responsible for coordinating and executing backup tasks to ensure that the data is safely backed up to the specified storage volume, thereby ensuring the integrity and availability of the data.

[0071] In one embodiment, see Figure 4 , the above-mentioned step S130 may include steps S131 to S134.

[0072] S131, initiating a backup job task and allocating a storage volume from a storage server;

[0073] S132. Analyze the topology of the TiDB database cluster, automatically identify the TiKV node and the control node, and send the backup job tasks and parameters to the client of the control node, so that the client of the control node can obtain the consistency time point through pd-ctl, suspend GC and scheduling, and feedback the job status.

[0074] In this embodiment, after the CDM backup platform starts the backup job, it first allocates a storage volume from the storage server for this backup. Next, the CDM backup platform analyzes the topology of the TiDB database cluster, automatically identifies and sends the job tasks and backup parameters to the corresponding client, including the backup type and storage volume information, etc.; after the client of the control node receives the job task, it calls the TiDB cluster management tool (such as pd-ctl) to obtain the consistency time point (TSO), which is used to restore the cluster to this time point and suspend the GC and scheduling functions. After the task is successfully executed, the CDM platform sends the backup job to the clients of all TiKV nodes and executes it in parallel.

[0075] S133. After the control node job is successfully executed, the backup job task is sent to the clients of the TiKV nodes of all TiDB databases, so that the clients of the TiKV nodes can take a snapshot of the file system through the CBT component of Yunxinda and copy the snapshot data to the storage volume.

[0076] In this embodiment, when the control node job is successfully executed, the backup job task is sent to the client of the TiKV node of all TiDB databases, so that the client of the TiKV node obtains the directory of each TiKV data node and analyzes the corresponding file system, uses Yunxinda's CBT component to snapshot the file system, and copies the snapshot data to the storage volume.

[0077] Specifically, after receiving the backup task, the client of each TiKV node obtains the data node directory and analyzes the file system, uses Yunxinda's CBT component to snapshot the file system, and then copies the snapshot data to the specified storage volume on the backup storage server.

[0078] S134. After all TiKV nodes complete the snapshot, the client of the scheduling control node re-enables the GC and scheduling functions.

[0079] In this embodiment, after all TiKV nodes complete the snapshot, the CDM platform schedules the client of the control node to start the GC and scheduling functions. After completing the snapshot data copy task, each node client deletes the snapshot and completes the node backup job.

[0080] TiDB's GC stands for Garbage Collection. The GC process aims to clean up old data and reduce its impact on performance. It mainly includes four steps: calculating GC safepoint, resolving locks, deleting continuous ranges, and synchronizing GCsafepoint to other components of the cluster.

[0081] TiDB's scheduling refers to PD (Placement Driver) scheduling, which includes the distribution and migration of data and replicas, node health check and fault recovery, hotspot scheduling, cluster topology awareness, etc.

[0082] The CDM platform periodically queries the backup progress information reported by all TiKV node clients until all nodes report that the backup is successful, thus completing the backup job.

[0083] The CDM backup platform can automatically execute the backup process, including storage volume allocation, job task scheduling, and monitoring progress, reducing the need for manual operations and improving the efficiency and accuracy of operations. Through the automated backup process, sufficient storage space for backup data is ensured, and the control node and the TiKV node client coordinate the execution of backup tasks to ensure that the data is safely backed up to the specified storage volume, thereby maintaining the integrity and availability of the data. The TiDB cluster management tool is used to obtain the consistent time point (TSO), suspend the GC (Garbage Collection) and scheduling functions during the backup process, and ensure the consistency and recoverability of the backup data, which is very important for data recovery and fault recovery. The CDM platform can execute backup tasks in parallel and process the data snapshots of TiKV nodes in a distributed manner, which improves the speed and efficiency of backup operations while reducing the impact on cluster performance. The CDM platform regularly queries the backup progress information of all TiKV node clients to ensure that the backup process can be monitored in real time and the backup status can be fed back in time, thereby ensuring the smooth completion of the backup task and the security of the data.

[0084] In summary, this automated backup process based on the CDM backup platform not only improves the efficiency and reliability of backup operations, but also ensures the security and consistency of TiDB distributed database data, providing important technical support for system management and data recovery.

[0085] S140. Select a backup data set, and use snapshot technology and multi-node collaboration functions to restore the backup data set to a specified target cluster.

[0086] In this embodiment, when the backup administrator triggers an instant recovery request, the CDM backup platform will take snapshots of the previously backed-up dataset volumes. These snapshot volumes will be mounted to the client of the TiKV node using the NFS or ISCSI protocol. The control node is responsible for clearing, starting and stopping the cluster, and performing recovery consistency processing.

[0087] In one embodiment, see Figure 4 , the above-mentioned step S140 may include steps S141 to S146.

[0088] S141. Select the data set to be restored, specify the target cluster, and start the restoration operation;

[0089] S142, the job scheduling component automatically analyzes whether the target cluster meets the recovery conditions, parses the control node and TiKV node of the target cluster, and confirms whether the corresponding client has been installed on the control node and TiKV node;

[0090] S143. When the target cluster meets the recovery conditions and the corresponding client has been installed on the control node and TiKV node, a snapshot of the data set to be restored is taken;

[0091] S144, sending a job to the client of the control node, so that the client of the control node clears the target cluster and starts the empty PD node, and feedbacks the status after success;

[0092] S145, mount the data in the restored storage volume to all TiKV nodes, connect the mounted data directory to the data directory of the TiKV node through a soft link, and modify the directory permissions;

[0093] S146. When the clients of all TiKV nodes successfully mount the file system, the scheduling component sends a startup job for restoring the target cluster to the client of the control node, so that the client of the control node executes the startup job for restoring the target cluster until the client of the control node reports a successful recovery.

[0094] In this embodiment, when the clients of all TiKV nodes successfully mount the file system, the scheduling component sends a startup job for restoring the target cluster to the client of the control node, so that the client of the control node switches the PD node to recovery mode, starts all TiKV services, stops the cluster scheduling function, uses the BR tool to restore the target cluster to the consistency time point at the time of backup, exits the recovery mode, and restarts the TiDB database cluster.

[0095] First, the backup administrator selects the data set to be restored, specifies the target cluster, and initiates the recovery job; the job scheduling component (Director) of the CDM backup platform automatically analyzes and determines whether the target cluster meets the recovery conditions. At the same time, it parses the control node and TiKV node of the target cluster and confirms whether the client has been installed on the corresponding node; the CDM platform first takes a snapshot of the backed-up data set to ensure that the backup data can be used in a variety of recovery scenarios; after the recovery data is ready, the CDM platform sends jobs to the client of the control node. These jobs include clearing the target cluster, starting the empty PD node, and reporting the status to the CDM platform after the operation is successful; the CDM platform mounts the restored storage volume data to all TiKV nodes in ISCSI or NFS mode, connects the mounted data directory to the TiKV data directory through soft links, and adjusts the file directory permissions; after all TiKV clients successfully mount the file system, the CDM platform scheduling component continues to send jobs to restore the cluster to the client of the control node. These operations include putting the PD into recovery mode, starting all TiKV services, stopping the cluster scheduling function, and using the BR tool to restore the cluster to the consistent time point (TSO) obtained during the backup. This process ensures that the TiKV component rolls back to the specified time point based on the log on all nodes to achieve global consistency. After completing the recovery mode, the entire TiDB cluster restarts. If all the above steps are executed successfully, the CDM platform returns the status information of successful recovery; the CDM platform waits for the control node client to report the successful recovery information to complete the entire recovery task.

[0096] These steps ensure smooth execution and consistency when restoring data on the CDM backup platform.

[0097] The method of this embodiment is based on CDM snapshot backup, and is deeply optimized for the recovery needs of the TiDB database, aiming to improve backup efficiency and meet the requirements of various actual business scenarios. By adopting the disk snapshot backup method, data packaging and excessive overhead on resources such as the TiDB database CPU are avoided. At the same time, the backup data is stored in the original format, which reduces the decompression processing time during recovery and improves recovery efficiency and speed. In addition, the use of data block replication for backup significantly improves backup efficiency compared to small file copying, and avoids problems such as table locks and latches that may occur during internal database processing.

[0098] Through virtual copy technology, a single copy of backup data can be used in multiple scenarios, supporting the rapid generation of new database clusters. Regardless of the size of the data, recovery can be completed in minutes, meeting customers' needs for rapid recovery and rapid data supply in the event of a failure. In addition, CBT backup technology eliminates the need to mount the NFS file system or prepare an object storage environment during the backup process, greatly reducing the difficulty of backup operations. The backup platform automatically allocates storage resources and schedules jobs, records the consistency points of TiDB data, and effectively manages stop, start, and GC functions, improving the efficiency and stability of overall backup management.

[0099] Specifically, the disk snapshot backup method avoids data packaging, and the compression function is handled by the media service, which significantly reduces the consumption of TiDB database CPU and other resources, and has minimal impact on the production environment. The backup data is stored in the original format, and no decompression is required during recovery, which greatly improves the recovery efficiency and speed. The backup is performed by data block replication, and the backup efficiency is significantly better than the traditional small file copy method. The internal database processing is avoided during the backup process, which effectively prevents table locking, latching and other behaviors, and ensures the stability and reliability of the backup. Multiple copies are generated by the virtual copy service and directly mounted to the server. One copy of the backup data can be used in multiple scenarios, which improves the flexibility and diversified application of the data. The incremental synthesis method is used for backup, which reduces the waste of storage resources and supports recovery through incremental data after deleting the full backup. CBT backup technology does not require mounting the NFS file system or object storage environment, which greatly simplifies the backup configuration and operation difficulty. The backup platform automatically allocates storage resources, automatically schedules jobs, records the consistency points of TiDB data, supports stop, start and GC functions, and ensures the efficiency and stability of the backup process. The data transmission from the backup client to the storage medium adopts streaming transmission, which further improves the backup speed and efficiency. It supports fast recovery at the minute level, regardless of the amount of data, to meet the customer's emergency recovery needs in the event of a failure. It supports database recovery to a specified consistency point to ensure the integrity and accuracy of the restored data. These features make this backup solution not only suitable for efficient backup of large-scale TiDB databases, but also for rapid data recovery in the face of failures, ensuring business continuity and data security.

[0100] In summary, this solution greatly simplifies the management and recovery process of the TiDB database through innovative design and efficient technical architecture, and significantly improves the reliability and operational efficiency of the system.

[0101] The above-mentioned TiDB distributed database is based on the CDM technology for fast data backup and recovery. The CDM backup platform creates storage volumes and configures backup policies. After the client is deployed, when the operation request is a backup request, the system allocates storage volumes and schedules tasks, and monitors the backup progress in real time. For non-backup requests but involving TiDB data recovery, the platform uses snapshot technology and multi-node collaboration functions to select and restore backup data sets to the specified target cluster to achieve fast and consistent recovery, significantly simplifying backup and recovery operations and improving efficiency. It is particularly suitable for the strict disaster recovery time target requirements of the financial industry.

[0102] Figure 6 1 is a schematic block diagram of a TiDB distributed database fast data backup and recovery device 300 based on CDM technology provided by an embodiment of the present invention. Figure 6 As shown, corresponding to the above TiDB distributed database based on CDM technology fast data backup and recovery method, the present invention also provides a TiDB distributed database based on CDM technology fast data backup and recovery device 300. The TiDB distributed database based on CDM technology fast data backup and recovery device 300 includes a unit for executing the above TiDB distributed database based on CDM technology fast data backup and recovery method, and the device can be configured in a server. Specifically, please refer to Figure 6 The TiDB distributed database fast data backup and recovery device 300 based on CDM technology includes a backup preparation unit 301, a request acquisition unit 302, a backup unit 303 and a recovery unit 304.

[0103] The backup preparation unit 301 is used to create storage volumes, configure backup policies and deploy clients on the CDM backup platform; the request acquisition unit 302 is used to obtain operation requests; the backup unit 303 is used to back up TiDB distributed database data by allocating storage volumes, scheduling job tasks and monitoring progress in real time; the recovery unit 304 is used to select backup data sets and use snapshot technology and multi-node collaboration functions to restore the backup data sets to the specified target cluster.

[0104] In one embodiment, the backup preparation unit 301 includes a creation subunit, a configuration subunit, and a deployment subunit.

[0105] The creation subunit is used to create storage volumes on the CDM backup platform. The configuration subunit is used to configure the database backup strategy on the CDM backup platform and select the data backup format as the original format. The deployment subunit is used to deploy the client on the control node of the TiDB distributed database and set it as the master node, and deploy the client on all TiKV nodes and set them as slave nodes.

[0106] In one embodiment, the backup unit 303 includes an allocation subunit, an identification subunit, a sending subunit, and a scheduling subunit.

[0107] The allocation subunit is used to initiate a backup job task and allocate storage volumes from the storage server; the identification subunit is used to analyze the topology of the TiDB database cluster, automatically identify TiKV nodes and control nodes, and send the backup job task and parameters to the client of the control node, so that the client of the control node can obtain the consistency time point through pd-ctl, suspend GC and scheduling, and feedback the job status; the sending subunit is used to send the backup job task to the clients of the TiKV nodes of all TiDB databases after the control node job is successfully executed, so that the clients of the TiKV nodes can take a snapshot of the file system through the CBT component of Yunxinda and copy the snapshot data to the storage volume; the scheduling subunit is used to schedule the client of the control node to restart the GC and scheduling functions after all TiKV nodes have completed the snapshot.

[0108] In one embodiment, the sending subunit is used to send the backup job task to the clients of the TiKV nodes of all TiDB databases after the control node job is successfully executed, so that the client of the TiKV node can obtain the directory of each TiKV data node and analyze the corresponding file system, use the CBT component of Yunxinda to take a snapshot of the file system, and copy the snapshot data to the storage volume.

[0109] In one embodiment, the reply unit includes a selection subunit, an analysis subunit, a snapshot subunit, a sending subunit, a hanging subunit and a starting subunit.

[0110] The selection subunit is used to select the data set to be restored, specify the target cluster, and start the recovery job; the analysis subunit is used by the job scheduling component to automatically analyze whether the target cluster meets the recovery conditions, parse the control node and TiKV node of the target cluster, and confirm whether the corresponding client has been installed on the control node and TiKV node; the snapshot subunit is used to take a snapshot of the data set to be restored when the target cluster meets the recovery conditions and the corresponding client has been installed on the control node and TiKV node; the sending subunit is used to send jobs to the client of the control node so that the client of the control node clears the target cluster and starts the empty PD node, and feedbacks the status after success; the mounting subunit is used to mount the data in the restored storage volume to all TiKV nodes, connect the mounted data directory to the data directory of the TiKV node through a soft link, and modify the directory permissions; the startup subunit is used when the clients of all TiKV nodes successfully mount the file system, and the scheduling component sends the startup job of restoring the target cluster to the client of the control node so that the client of the control node executes the startup job of restoring the target cluster until the client of the control node reports the information of successful recovery.

[0111] In one embodiment, the startup subunit is used to send a startup job for restoring the target cluster to the client of the control node when the clients of all TiKV nodes successfully mount the file system, so that the client of the control node switches the PD node to recovery mode, starts all TiKV services, stops the cluster scheduling function, uses the BR tool to restore the target cluster to the consistency time point at the time of backup, exits the recovery mode, and restarts the TiDB database cluster.

[0112] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned TiDB distributed database based on CDM technology fast data backup and recovery device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment. For the convenience and conciseness of the description, it will not be repeated here.

[0113] The TiDB distributed database fast data backup and recovery device 300 based on CDM technology can be implemented in the form of a computer program. The computer program can be used in Figure 7 Runs on the computer device shown.

[0114] See also Figure 7 , Figure 7 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0115] See also Figure 7The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0116] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, can enable the processor 502 to execute a TiDB distributed database fast data backup and recovery method based on CDM technology.

[0117] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0118] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a TiDB distributed database fast data backup and recovery method based on CDM technology.

[0119] The network interface 505 is used to communicate with other devices over the network. Figure 7 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:

[0121] Create storage volumes, configure backup policies, and deploy clients on the CDM backup platform; obtain operation requests; back up TiDB distributed database data by allocating storage volumes, scheduling job tasks, and monitoring progress in real time; select backup data sets, and use snapshot technology and multi-node collaboration functions to restore backup data sets to the specified target cluster.

[0122] In one embodiment, the processor 502 implements the following steps when implementing the steps of creating a storage volume, configuring a backup policy, and deploying a client on the CDM backup platform:

[0123] Create storage volumes on the CDM backup platform; configure the database backup policy on the CDM backup platform and select the original format for data backup; deploy the client on the control node of the TiDB distributed database and set it as the master node, and deploy the client on all TiKV nodes and set them as slave nodes.

[0124] The backup strategy includes setting the backup initiation time and the expiration time of the backup data set.

[0125] In one embodiment, when implementing the steps of backing up TiDB distributed database data by allocating storage volumes, scheduling job tasks, and real-time monitoring progress, the processor 502 specifically implements the following steps:

[0126] Initiate a backup job task and allocate storage volumes from the storage server; analyze the topology of the TiDB database cluster, automatically identify TiKV nodes and control nodes, and send the backup job task and parameters to the client of the control node, so that the client of the control node can obtain the consistency time point through pd-ctl, suspend GC and scheduling, and feedback the job status; when the control node job is successfully executed, send the backup job task to the clients of the TiKV nodes of all TiDB databases, so that the clients of the TiKV nodes can take a snapshot of the file system through Yunxinda's CBT component and copy the snapshot data to the storage volume; when all TiKV nodes complete the snapshot, schedule the client of the control node to restart the GC and scheduling functions.

[0127] In one embodiment, when the processor 502 implements the step of sending the backup job task to the clients of the TiKV nodes of all TiDB databases after the control node job is successfully executed, so that the clients of the TiKV nodes can take a snapshot of the file system through the CBT component of Yunxinda and copy the snapshot data to the storage volume, the following steps are specifically implemented:

[0128] When the control node job is successfully executed, the backup job task is sent to the clients of the TiKV nodes of all TiDB databases, so that the TiKV node clients can obtain the directory of each TiKV data node and analyze the corresponding file system, use Yunxinda's CBT component to snapshot the file system, and copy the snapshot data to the storage volume.

[0129] In one embodiment, when the processor 502 implements the step of selecting a backup data set and restoring the backup data set to a specified target cluster using snapshot technology and multi-node collaboration function when the operation request is TiDB distributed database data recovery, the processor 502 specifically implements the following steps:

[0130] Select the data set to be restored, specify the target cluster, and start the recovery job; the job scheduling component automatically analyzes whether the target cluster meets the recovery conditions, parses the control node and TiKV node of the target cluster, and confirms whether the corresponding client has been installed on the control node and TiKV node; when the target cluster meets the recovery conditions and the corresponding client has been installed on the control node and TiKV node, take a snapshot of the data set to be restored; send a job to the client of the control node so that the client of the control node clears the target cluster and starts the empty PD node, and feedbacks the status after success; mount the data in the restored storage volume to all TiKV nodes, connect the mounted data directory to the data directory of the TiKV node through a soft link, and modify the directory permissions; when the clients of all TiKV nodes successfully mount the file system, the scheduling component sends the startup job of restoring the target cluster to the client of the control node so that the client of the control node executes the startup job of restoring the target cluster until the client of the control node reports the successful recovery information.

[0131] In one embodiment, when the processor 502 implements the step of sending a startup job of restoring the target cluster to the client of the control node when the clients of all TiKV nodes successfully mount the file system, so that the client of the control node executes the startup job of restoring the target cluster until the client of the control node reports the information of successful recovery, the processor 502 specifically implements the following steps:

[0132] When the clients of all TiKV nodes successfully mount the file system, the scheduling component sends a startup job to restore the target cluster to the client of the control node, so that the client of the control node switches the PD node to recovery mode, starts all TiKV services, stops the cluster scheduling function, uses the BR tool to restore the target cluster to the consistency time point at the time of backup, exits recovery mode, and restarts the TiDB database cluster.

[0133] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0134] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0135] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0136] Create storage volumes, configure backup policies, and deploy clients on the CDM backup platform; obtain operation requests; back up TiDB distributed database data by allocating storage volumes, scheduling job tasks, and monitoring progress in real time; select backup data sets, and use snapshot technology and multi-node collaboration functions to restore backup data sets to the specified target cluster.

[0137] In one embodiment, when the processor executes the computer program to implement the steps of creating a storage volume, configuring a backup strategy, and deploying a client on the CDM backup platform, the processor specifically implements the following steps:

[0138] Create storage volumes on the CDM backup platform; configure the database backup policy on the CDM backup platform and select the original format for data backup; deploy the client on the control node of the TiDB distributed database and set it as the master node, and deploy the client on all TiKV nodes and set them as slave nodes.

[0139] The backup strategy includes setting the backup initiation time and the expiration time of the backup data set.

[0140] In one embodiment, when the processor executes the computer program to implement the steps of backing up the TiDB distributed database data by allocating storage volumes, scheduling job tasks, and real-time monitoring progress, the processor specifically implements the following steps:

[0141] Initiate a backup job task and allocate storage volumes from the storage server; analyze the topology of the TiDB database cluster, automatically identify TiKV nodes and control nodes, and send the backup job task and parameters to the client of the control node, so that the client of the control node can obtain the consistency time point through pd-ctl, suspend GC and scheduling, and feedback the job status; when the control node job is successfully executed, send the backup job task to the clients of the TiKV nodes of all TiDB databases, so that the clients of the TiKV nodes can take a snapshot of the file system through Yunxinda's CBT component and copy the snapshot data to the storage volume; when all TiKV nodes complete the snapshot, schedule the client of the control node to restart the GC and scheduling functions.

[0142] In one embodiment, when the processor executes the computer program to implement the step of sending the backup job task to the clients of the TiKV nodes of all TiDB databases after the control node job is successfully executed, so that the clients of the TiKV nodes can take a snapshot of the file system through the CBT component of Yunxinda, and copy the snapshot data to the storage volume, the following steps are specifically implemented:

[0143] When the control node job is successfully executed, the backup job task is sent to the clients of the TiKV nodes of all TiDB databases, so that the TiKV node clients can obtain the directory of each TiKV data node and analyze the corresponding file system, use Yunxinda's CBT component to snapshot the file system, and copy the snapshot data to the storage volume.

[0144] In one embodiment, when the processor executes the computer program to implement the step of selecting a backup data set and restoring the backup data set to a specified target cluster using snapshot technology and multi-node collaboration function when the operation request is TiDB distributed database data recovery, the processor specifically implements the following steps:

[0145] Select the data set to be restored, specify the target cluster, and start the restore job;

[0146] The job scheduling component automatically analyzes whether the target cluster meets the recovery conditions, parses the control node and TiKV node of the target cluster, and confirms whether the corresponding client has been installed on the control node and TiKV node; when the target cluster meets the recovery conditions and the corresponding client has been installed on the control node and TiKV node, a snapshot of the data set to be restored is taken; a job is sent to the client of the control node so that the client of the control node clears the target cluster and starts the empty PD node, and feedbacks the status after success; the data in the restored storage volume is mounted to all TiKV nodes, the mounted data directory is connected to the data directory of the TiKV node through a soft link, and the directory permissions are modified; when the clients of all TiKV nodes successfully mount the file system, the scheduling component sends the startup job of restoring the target cluster to the client of the control node so that the client of the control node executes the startup job of restoring the target cluster until the client of the control node reports the successful recovery information.

[0147] In one embodiment, when the processor executes the computer program to implement the step of sending a startup job of restoring the target cluster to the client of the control node when the clients of all TiKV nodes successfully mount the file system, so that the client of the control node executes the startup job of restoring the target cluster until the client of the control node reports information of successful recovery, the following steps are specifically implemented:

[0148] When the clients of all TiKV nodes successfully mount the file system, the scheduling component sends a startup job to restore the target cluster to the client of the control node, so that the client of the control node switches the PD node to recovery mode, starts all TiKV services, stops the cluster scheduling function, uses the BR tool to restore the target cluster to the consistency time point at the time of backup, exits recovery mode, and restarts the TiDB database cluster.

[0149] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0150] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0151] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0152] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0154] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. The TiDB distributed database fast data backup and recovery method based on CDM technology is characterized by: include: Create storage volumes, configure backup policies, and deploy clients on the CDM backup platform; Get operation requests to back up TiDB distributed database data by allocating storage volumes, scheduling job tasks, and monitoring progress in real time; Select a backup dataset and use snapshot technology and multi-node collaboration to restore the backup dataset to the specified target cluster.

2. According to claim 1, the TiDB distributed database fast data backup and recovery method based on CDM technology is characterized in that: Creating storage volumes, configuring backup policies, and deploying clients on the CDM backup platform include: Create a storage volume on the CDM backup platform; Configure the database backup policy on the CDM backup platform and select the original format for data backup; Deploy the client on the control node of the TiDB distributed database and set it as the master node. Deploy the client on all TiKV nodes and set them as slave nodes.

3. According to claim 2, the TiDB distributed database fast data backup and recovery method based on CDM technology is characterized in that: The backup strategy includes setting the backup initiation time and the expiration time of the backup data set.

4. According to claim 2, the TiDB distributed database fast data backup and recovery method based on CDM technology is characterized in that: The method of backing up TiDB distributed database data by allocating storage volumes, scheduling job tasks, and real-time monitoring progress includes: Initiate a backup job and allocate storage volumes from the storage server; Analyze the topology of the TiDB database cluster, automatically identify TiKV nodes and control nodes, and send backup job tasks and parameters to the client of the control node. The client of the control node can obtain the consistency time point through pd-ctl, suspend GC and scheduling, and feedback the job status. When the control node job is successfully executed, the backup job task is sent to the clients of the TiKV nodes of all TiDB databases, so that the clients of the TiKV nodes can take a snapshot of the file system through the CBT component of Cloud Trust and copy the snapshot data to the storage volume; When all TiKV nodes complete the snapshot, the client of the scheduling control node re-enables the GC and scheduling functions.

5. According to claim 4, the TiDB distributed database fast data backup and recovery method based on CDM technology is characterized in that: When the control node job is successfully executed, the backup job task is sent to the client of the TiKV node of all TiDB databases, so that the client of the TiKV node can take a snapshot of the file system through the CBT component of Yunxinda and copy the snapshot data to the storage volume, including: When the control node job is successfully executed, the backup job task is sent to the clients of the TiKV nodes of all TiDB databases, so that the TiKV node clients can obtain the directory of each TiKV data node and analyze the corresponding file system, use Yunxinda's CBT component to snapshot the file system, and copy the snapshot data to the storage volume.

6. The TiDB distributed database fast data backup and recovery method based on CDM technology according to claim 1 is characterized in that: When the operation request is for TiDB distributed database data recovery, a backup data set is selected, and the backup data set is restored to the specified target cluster using snapshot technology and multi-node collaboration functions, including: Select the data set to be restored, specify the target cluster, and start the restore job; The job scheduling component automatically analyzes whether the target cluster meets the recovery conditions, parses the control node and TiKV node of the target cluster, and confirms whether the corresponding client has been installed on the control node and TiKV node; When the target cluster meets the recovery conditions and the corresponding client has been installed on the control node and TiKV node, take a snapshot of the data set to be restored; Send a job to the client of the control node so that the client of the control node can clear the target cluster and start the empty PD node, and feedback the status after success; Mount the data in the restored storage volume to all TiKV nodes, connect the mounted data directory to the data directory of the TiKV node through a soft link, and modify the directory permissions; When the clients of all TiKV nodes successfully mount the file system, the scheduling component sends the startup job of restoring the target cluster to the client of the control node, so that the client of the control node executes the startup job of restoring the target cluster until the client of the control node reports the successful recovery information.

7. The TiDB distributed database fast data backup and recovery method based on CDM technology according to claim 6 is characterized in that: When the clients of all TiKV nodes successfully mount the file system, the scheduling component sends a startup job of restoring the target cluster to the client of the control node, so that the client of the control node executes the startup job of restoring the target cluster until the client of the control node reports the information of successful recovery, including: When the clients of all TiKV nodes successfully mount the file system, the scheduling component sends a startup job to restore the target cluster to the client of the control node, so that the client of the control node switches the PD node to recovery mode, starts all TiKV services, stops the cluster scheduling function, uses the BR tool to restore the target cluster to the consistency time point at the time of backup, exits recovery mode, and restarts the TiDB database cluster.

8. TiDB distributed database fast data backup and recovery device based on CDM technology, characterized by: include: The backup preparation unit is used to create storage volumes, configure backup policies, and deploy clients on the CDM backup platform. A request acquisition unit, used for acquiring an operation request; The backup unit is used to back up TiDB distributed database data by allocating storage volumes, scheduling job tasks, and monitoring progress in real time; The recovery unit is used to select a backup data set and restore the backup data set to a specified target cluster using snapshot technology and multi-node collaboration functions.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images