Data backup method, device and equipment and computer readable storage medium

Abstract

The invention discloses a data backup method, device and equipment and a computer readable storage medium, which are applied to the field of database backup, and are characterized in that when full backup is triggered, nodes in a cluster are coordinated by using a central tuner to perform global logic time consistency backup; the nodes comprise fragment nodes and configuration server nodes; and when incremental backup is triggered, intercepting and backing up an operation log and a storage engine check point log based on a time window overlapping algorithm by utilizing a backup agent of each fragment node. According to the invention, each fragment in the cluster is tuned by the central tuner to generate the consistent backup data, so that the problem that each fragment data cannot be recovered to the same logic time point during data recovery is solved; and collecting mixed logs by using each backup agent, and eliminating the risk of high-frequency instantaneous data loss of the operation logs through a window overlapping algorithm.

Description

Technical Field

[0001] The present invention relates to the field of database backup, and in particular to a data backup method, device, equipment and computer-readable storage medium. Background Art

[0002] MongoDB is a distributed file-based database that sits somewhere between relational and non-relational databases. Sharded clusters are MongoDB's cluster architecture, allowing you to scale database capacity horizontally by increasing the number of shards as business data grows.

[0003] Data backup is a common database reliability measure. By backing up database data to a remote location, in the event of data loss or corruption in an unexpected scenario, the backup data can be quickly retrieved and restored to the database. Because each shard in MongoDB is an independent storage unit, backups are performed independently. This prevents data from multiple shards from being restored to the same point in time during backup and restore, leading to data unavailability. Furthermore, incremental backup solutions based on the oplog (operation log) are prone to time window blind spots, potentially missing some frequently occurring data.

[0004] It can be seen that how to coordinate the shards to generate consistent backup data and how to avoid high-frequency data loss during backup are technical problems that technical personnel in this field urgently need to solve. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a data backup method, device, equipment and computer-readable storage medium, which solves the problems in the prior art of how to coordinate each shard to generate consistent backup data and how to avoid high-frequency data loss during backup.

[0006] To solve the above technical problems, the present invention provides a data backup method, comprising:

[0007] When a full backup is triggered, the central tuner is used to coordinate the nodes in the cluster to perform a global logical time consistency backup; the nodes include each shard node and the configuration server node;

[0008] When an incremental backup is triggered, the backup agent of each shard node is used to intercept and back up the operation log and the storage engine checkpoint log based on a time window overlapping algorithm.

[0009] On the one hand, triggering a full backup includes:

[0010] Using the central tuner to monitor changes in metadata on the configuration server node in real time;

[0011] When it is detected that the cluster is scaled in or out or data is migrated, the full backup is triggered.

[0012] On the one hand, a central tuner is used to coordinate the global logical time consistency backup of each node in the cluster, including:

[0013] Obtaining, through the central tuner, a current operation log timestamp of each node in the cluster, and selecting a global maximum timestamp from a plurality of the current operation log timestamps as a global checkpoint timestamp;

[0014] Through the backup agent of each node, the operation log is intercepted according to the global checkpoint timestamp, and the storage engine interface is called to generate a storage engine consistency snapshot, and the storage engine consistency snapshot and the intercepted operation log are uploaded to the backup storage system.

[0015] On the one hand, obtaining the current operation log timestamp of each node in the cluster through the central tuner includes:

[0016] Sending a snapshot preparation instruction to all the nodes in the cluster through the central tuner;

[0017] When each of the nodes receives the snapshot preparation instruction, it suspends the data shard migration operation, records the current operation log timestamp and sends it to the central tuner.

[0018] On the one hand, the backup agent of each shard node is used to intercept and back up the operation log and storage engine checkpoint log based on the time window overlapping algorithm, including:

[0019] The central tuner calculates the collection time window of each shard node based on the busyness of each shard node and sends it to the corresponding shard node; the collection time window consists of a reference time collection window and an overlap time collection window;

[0020] Each of the shard nodes receives the corresponding collection time window, and through the backup agent, periodically intercepts the operation log and the storage engine checkpoint log based on the respective collection time window, and uploads them to the backup storage system.

[0021] On the one hand, the central tuner calculates the collection time window of each shard node based on the busyness of each shard node, and sends it to the corresponding shard node, including:

[0022] The backup agent of each shard node periodically collects the write operation frequency and write data bandwidth of the respective shards and sends the data to the central tuner;

[0023] The central tuner obtains the write operation frequency and the write data bandwidth of all shards, calculates the acquisition time window of each shard node, and sends it to the corresponding shard node.

[0024] On the one hand, periodically intercepting the operation log and the storage engine checkpoint log based on the respective collection time windows and uploading them to the backup storage system includes:

[0025] Taking the previous acquisition end point as a reference, the overlapping time acquisition window is shifted forward as the starting point of this acquisition; the previous acquisition end point is generated based on the trigger of the reference time acquisition window;

[0026] The operation log and the storage engine checkpoint log are intercepted from the starting point of the current collection and uploaded to the backup storage system.

[0027] The present invention also provides a data backup device, comprising:

[0028] A full backup module is used to coordinate the global logical time consistency backup of each node in the cluster using the central tuner when a full backup is triggered; the nodes include each shard node and the configuration server node;

[0029] The incremental backup module is used to intercept and back up the operation log and storage engine checkpoint log based on the time window overlapping algorithm by using the backup agent of each shard node when the incremental backup is triggered.

[0030] The present invention also provides a data backup device, comprising:

[0031] memory for storing computer programs;

[0032] A processor is used to implement the above-mentioned data backup method when executing the computer program.

[0033] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the above-mentioned data backup method is implemented.

[0034] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above-mentioned data backup method when executed by a processor.

[0035] As can be seen from the above technical solution, the present invention utilizes a central tuner to coordinate the nodes in the cluster to perform a global logical time consistency backup when responding to a full backup; the nodes include each shard node and a configuration server node; and when responding to a triggered incremental backup, the backup agent of each shard node is used to intercept and back up the operation log and storage engine checkpoint log based on a time window overlapping algorithm. The beneficial effects of the present invention are: during a full backup, the central tuner tunes each shard in the cluster to generate consistent backup data, which ensures that the data of each shard can be restored to the same logical time point during data recovery; during an incremental backup, each backup agent collects mixed logs, and the risk of losing high-frequency instantaneous data in the operation log is eliminated through a window overlapping algorithm.

[0036] In addition, the present invention also provides a data backup device, equipment and computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 This is a typical architecture diagram of an existing sharded cluster;

[0039] Figure 2 A flowchart of a data backup method provided by an embodiment of the present invention;

[0040] Figure 3 An optimized architecture diagram of a sharded cluster provided in an embodiment of the present invention;

[0041] Figure 4 A schematic structural diagram of a data backup device provided by an embodiment of the present invention;

[0042] Figure 5 A schematic structural diagram of a data backup device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] First, some terms involved in this invention are analyzed:

[0045] MongoDB: It is a database based on distributed file storage and is a product between relational databases and non-relational databases.

[0046] Write QPS (write operation frequency): QPS refers to the number of database queries per second, and write QPS refers to the number of database write requests per second.

[0047] Write data bandwidth: The amount of data written to the database per second.

[0048] Oplog: The operation log is a special collection in MongoDB that records changes in all database operations.

[0049] WiredTiger Checkpoint: WiredTiger is the default storage engine for MongoDB. Checkpoints are a key feature of WiredTiger. Their primary purpose is to write modified data in memory to data files, ensuring data persistence and reducing recovery time in the event of a database failure. A checkpoint is equivalent to a log, referred to in this example as the storage engine checkpoint log, which records changes to related data files since the last checkpoint.

[0050] Time window blind spot: There is a time difference (usually milliseconds to seconds) between when data is written to memory and when it is actually persisted to disk. If a failure occurs during this time, this part of the data will be lost.

[0051] High-frequency write scenarios: When the write rate is extremely high (e.g., tens of thousands of times per second), a large amount of data that is not written to disk accumulates in the memory buffer.

[0052] MongoDB database is widely used in social networking, games, live video and other scenarios. Sharding cluster is the cluster architecture of MongoDB, such as Figure 1 , Figure 1 This diagram shows a typical architecture of an existing sharded cluster. As business data increases, database capacity can be scaled out by adding shards. Within a shard, a single master node (Primary) and multiple slave nodes (Secondary) can be deployed to ensure the reliability of each shard. The config server node is responsible for storing database metadata, while the shard nodes store the actual business data.

[0053] Data backup is a common database reliability measure. By backing up database data to a remote location, database data loss or corruption can be quickly retrieved and restored to the database in the event of anomalies. Since each shard in a sharded cluster is an independent storage unit, if backups are performed independently on each shard, ensuring that data from multiple shards is restored to the same point in time (the database's logical timestamp) during backup recovery is a critical issue. Furthermore, to improve database backup efficiency, a data backup solution that combines full and incremental backups is often used. MongoDB database incremental backups typically use an oplog-based solution, but oplog-based incremental recovery solutions are prone to time window blind spots. If frequently written data has not yet been persisted from memory to disk, this data may be lost.

[0054] Therefore, in order to solve the above-mentioned problem of multi-shard backup consistency and high-frequency data loss in backup, the present invention provides a backup method. Figure 2 , Figure 2 A flowchart of a data backup method provided by an embodiment of the present invention. The method may include:

[0055] S101: When a full backup is triggered, the central tuner is used to coordinate the nodes in the cluster to perform a global logical time consistency backup; the nodes include each shard node and the configuration server node.

[0056] In this embodiment, the mongos node serves as the access point, primarily forwarding requests and not storing actual business data. Shard nodes, on the other hand, store substantial business data, and config server nodes also store substantial data, namely metadata. Therefore, this embodiment requires a global logical time-consistent backup of each shard node and config server node to ensure backup data consistency.

[0057] This embodiment does not limit the triggering conditions for the full backup. For example, it can be a periodic trigger; or it can be a conditional trigger. It is understandable that when a new shard is expanded or reduced in the cluster, a full backup must be triggered on the new shard before an incremental backup can be performed subsequently. However, the current backup system cannot perceive this process, and requires the administrator to manually identify and operate it, which will bring great inconvenience to operation and maintenance. Therefore, the full backup in this embodiment can be actively triggered by the backup system. The specific process includes: using the central tuner to monitor the changes in metadata on the configuration server nodes in real time; when it is detected that the cluster has expanded or reduced or data has migrated, a full backup is actively triggered. In other words, this embodiment monitors changes in cluster metadata through the central tuner, and actively triggers a full backup when the metadata changes, thereby achieving dynamic perception of cluster shards.

[0058] Furthermore, the above-mentioned use of the central tuner to coordinate the nodes in the cluster to perform global logical time consistency backup may include:

[0059] Step 11: Obtain the current operation log timestamp of each node in the cluster through the central tuner, and select the global maximum timestamp from multiple current operation log timestamps as the global checkpoint timestamp.

[0060] Specifically, this embodiment uses a central tuner to control backup operations on each node and selects the maximum oplog timestamp, T_max, as the global checkpoint timestamp. This use of the maximum oplog timestamp, T_max, as the global checkpoint timestamp essentially marks a "completely committed operation" on the database's logical timeline. From a data persistence perspective, this ensures that all committed operations are recorded, preventing data loss. From a consistency perspective, it provides a conflict-free time base for operations like backup, restore, and snapshots. From a performance perspective, clear time boundaries reduce redundant operations during backup and restore, improving efficiency.

[0061] Furthermore, the above-mentioned obtaining of the current operation log timestamp of each node in the cluster through the central tuner may include:

[0062] Step 111: Send snapshot preparation instructions to all nodes in the cluster through the central tuner;

[0063] Step 112: When each node receives the snapshot preparation instruction, it suspends the data shard migration operation, records the current operation log timestamp and sends it to the central tuner.

[0064] Specifically, the central tuner sends a prepareSnapshot command to each node. Each node pauses trunk (port aggregation) migration, pausing data shard migration operations, and records the current oplog timestamp, T_commit, and sends it to the central tuner. The central tuner waits for all nodes to return T_commit_i (1≤i≤n+1, where n is the number of nodes in the cluster) and calculates T_max, which is max(T_commit_0, T_commit_1, T_commit_n).

[0065] Step 12: Through the backup agent of each node, the operation log is intercepted according to the global checkpoint timestamp, and the storage engine interface is called to generate a storage engine consistency snapshot. The storage engine consistency snapshot and the intercepted operation log are uploaded to the backup storage system.

[0066] Specifically, each node's agent (backup agent) obtains T_max (i.e., the local checkpoint timestamp) from the central tuner, extracts the oplog based on T_max, and calls the WiredTiger API (storage engine interface) to generate a consistent storage engine snapshot at that point in time. A consistent storage engine snapshot is a complete image of the database's data at a specific point in time (including physical structures such as data files and indexes), equivalent to an "instantaneous copy of the full data." During database recovery, the central tuner controls data playback and rebuilds each node's data based on T_max. This embodiment extracts operation logs based on the global checkpoint timestamp and uploads them to the backup storage system to ensure operational consistency at the time the snapshot is generated (i.e., all operations up to that point in time have been committed). This snapshot may not need to be used directly during recovery.

[0067] S102: When an incremental backup is triggered, the backup agent of each shard node is used to intercept and back up the operation log and storage engine checkpoint log based on a time window overlap algorithm.

[0068] Specifically, this embodiment periodically collects hybrid logs, including oplogs (operation logs) and WT checkpoint logs (storage engine checkpoint logs). It also uses a time window overlap algorithm to eliminate data gaps, ensuring that instantaneous high-frequency data is not lost between incremental backups. Data can be restored to any point in time.

[0069] Furthermore, the above-mentioned use of the backup agent of each shard node to intercept and back up the operation log and storage engine checkpoint log based on the time window overlapping algorithm may specifically include:

[0070] Step 21: The central tuner calculates the collection time window of each shard node based on the busyness of each shard node and sends it to the corresponding shard node; the collection time window consists of a base time collection window and an overlap time collection window.

[0071] Specifically, the collection time window in this embodiment can be dynamically adjusted in real time based on business conditions. The busyness level in this embodiment is not limited. For example, it can be determined based on the frequency of data writes to the database. The base time collection window determines the primary time interval for hybrid log collection, i.e., the frequency of log collection triggering; the overlap time collection window defines the forward-expanding buffer range.

[0072] In this embodiment, the backup agent of each shard node periodically collects the write operation frequency and write data bandwidth of each shard and sends them to the central tuner; the central tuner calculates the collection time window of each shard node based on the obtained write operation frequency and write data bandwidth, and sends it to the corresponding shard nodes.

[0073] Specifically, the backup agent periodically collects the write QPS (denoted as wqps) and write bandwidth information (denoted as wkb) of each shard; the central tuner obtains the wqps of each shard i (1≤i≤n) and wkb i (1≤i≤n); calculate the collection time window size for mixed log collection of each shard. Set the initial size of the base time collection window to TW (for example, 600s) and the initial size of the overlapping time collection window to TWO (for example, 10s).

[0074] The benchmark time collection window size TW of each shard i The calculation method is:

[0075]

[0076] The overlapping time collection window size of each shard is TWO i The calculation method is:

[0077]

[0078] in, The factor that affects the window size for write QPS. It can be set as needed (greater than 0 and less than 1).

[0079] Furthermore, minimum and maximum thresholds are set for the reference time collection window so that the reference time collection window of each shard node is within the preset threshold range. This prevents resource waste caused by a too small reference time collection window, avoids the overall performance of the database being degraded due to ineffective high-frequency backups, and avoids the problem of excessive log recovery failures caused by a too large reference time collection window. When the write operation frequency exceeds A times the average write operation frequency of the cluster, the overlapping time collection window is increased to B times the reference time collection window. Here, A can be 3 and B can be 20%. This allows for real-time load intelligent adjustment of backup accuracy, strengthening protection at high frequencies and saving resources at low frequencies.

[0080] Step 22: Each shard node receives the corresponding collection time window, and through the backup agent, periodically intercepts the operation log and storage engine checkpoint log based on the respective collection time window, and uploads them to the backup storage system.

[0081] Specifically, the central tuner updates the reference time acquisition window size TW of each slice i and overlap time acquisition window size TWO i , and sent to the corresponding sharding node. So that the backup agent of each sharding node, based on TW i and TWO iThe oplog and WT Checkpoint logs are periodically captured and uploaded to the backup storage system. The oplog is an operation log captured from the last persistent markpoint to the current markpoint, serving as incremental operation capture. The oplog records all data changes that occurred between checkpoints, overwriting new operations that were continuously written during the forced disk flush. This embodiment, based on the operation log, adds the WT Checkpoint log and a time-overlapping window method to address the frequent data loss issue during incremental backups. The WT Checkpoint log is used to enforce a checkpoint before backup. It acts as a physical state anchor, forcing all pending data in memory to be flushed to disk, generating a persistent markpoint. This ensures that the corresponding physical data for each record in the oplog has been safely written to disk, eliminating the critical contradiction of "oplog records but no data on disk" and resolving the disconnect between the oplog and storage. This embodiment utilizes a time-overlapping window algorithm to eliminate gaps in disk flushing, achieving full lifecycle coverage of frequently written data, achieving zero data loss during the backup process, and meeting financial-grade data reliability requirements.

[0082] Furthermore, the above-mentioned periodic interception of operation logs and storage engine checkpoint logs based on their respective collection time windows and uploading to the backup storage system may include:

[0083] Step 221: Using the previous acquisition end point as a reference, the overlapping time acquisition window is shifted forward as the current acquisition start point; the previous acquisition end point is generated based on the triggering of the reference time acquisition window;

[0084] Step 222: intercept the operation log and storage engine checkpoint log from the current collection starting point, and upload them to the backup storage system.

[0085] For example: the end point of the previous acquisition is T1; the end point of this acquisition is: T2 = T1 + TW i (defined by the reference window), it can be seen that the previous acquisition end point T1 is based on the reference time acquisition window TW i Generated; actual collection interval: [T1 -TWO i , T2] (extending the overlapping area forward). In other words, the acquisition action is triggered with the reference time acquisition window as the period.

[0086] The data backup method provided by the embodiment of the present invention is applied. When responding to a full backup, a central tuner is used to coordinate the nodes in the cluster to perform a global logical time consistency backup; the nodes include each shard node and a configuration server node; when responding to a triggered incremental backup, the backup agent of each shard node is used to intercept and back up the operation log and the storage engine checkpoint log based on the time window overlapping algorithm. The beneficial effects of the present invention are: during a full backup, the central tuner tunes each shard in the cluster to generate consistent backup data, so that it can be ensured that the data of each shard can be restored to the same logical time point during data recovery; during an incremental backup, each backup agent collects a mixed log, and the risk of losing high-frequency instantaneous data in the operation log is eliminated through a window overlapping algorithm. The full and incremental backup based on this method can back up data at any time, and when a database failure occurs, the data can be restored to any time point after the start of the backup on demand.

[0087] You can refer to Figure 3 , Figure 3 The optimized architecture diagram of a shard cluster provided by an embodiment of the present invention. Among them, the central tuner cluster is deployed vigorously, responsible for issuing backup and recovery requests, coordinating the global logical time consistency backup of each shard, and also monitoring the changes in cluster metadata, actively triggering full backup when the metadata changes, so as to achieve dynamic perception of cluster shards; a backup agent is deployed in each shard, responsible for collecting oplog and WT Checkpoint logs, monitoring shard business loads, dynamically adjusting overlapping windows, and uploading or downloading backup data to the backup storage system. The backup process corresponds to Figure 3 1-3; the recovery process corresponds to Figure 3 Middle ac.

[0088] Accordingly, if you back up your data according to the above backup method, the data recovery process can refer to the following operations:

[0089] 1. Restore can be performed at any time: If you choose to restore at a certain time point t1, the central tuner will find the backup b1 closest to t1 from the backup records and execute 2. Restore can be performed according to the restore content: If you specify to backup at a certain backup b1, execute 2.

[0090] 2. If b1 is a full backup, the backup proxy downloads the backup data from the backup storage system. If it is an incremental backup, the backup proxy downloads both backup b1 and the subsequent full backup fb1 from the backup storage system.

[0091] 3. Restore the database based on the downloaded backup data. If you specify recovery at time point t1, the data will be replayed to t1 when the database is started.

[0092] The data backup device provided by an embodiment of the present invention is introduced below. The data backup device described below and the data backup method described above can be referenced to each other.

[0093] Please refer to Figure 4 , Figure 4 A schematic structural diagram of a data backup device provided in an embodiment of the present invention may include:

[0094] The full backup module 100 is used to coordinate the nodes in the cluster to perform a global logical time consistency backup using the central tuner when a full backup is triggered; the nodes include each shard node and the configuration server node;

[0095] The incremental backup module 200 is used to intercept and back up the operation log and storage engine checkpoint log based on the time window overlapping algorithm by using the backup agent of each shard node when the incremental backup is triggered.

[0096] Based on the above embodiment, the full backup module 100 may include:

[0097] A monitoring unit, configured to monitor changes in metadata on the configuration server node in real time using the central tuner;

[0098] The full backup triggering unit is used to trigger the full backup when it is detected that the cluster is scaled in or out or data is migrated.

[0099] Based on the above embodiment, the full backup module 100 may include:

[0100] a global checkpoint timestamp determining unit, configured to obtain, through the central tuner, the current operation log timestamp of each node in the cluster, and select a global maximum timestamp from a plurality of the current operation log timestamps as the global checkpoint timestamp;

[0101] The storage engine consistency snapshot generation unit is used to intercept the operation log according to the global checkpoint timestamp through the backup agent of each node, call the storage engine interface to generate a storage engine consistency snapshot, and upload the storage engine consistency snapshot and the intercepted operation log to the backup storage system.

[0102] Based on the above embodiment, the incremental backup module 200 may include:

[0103] an instruction sending unit, configured to send a snapshot preparation instruction to all the nodes in the cluster via the central tuner;

[0104] A timestamp recording unit is used to suspend the data shard migration operation when each node receives the snapshot preparation instruction, record the current operation log timestamp and send it to the central tuner.

[0105] Based on the above embodiment, the incremental backup module 200 may include:

[0106] a collection time window determination unit, configured to calculate, through the central tuner, a collection time window for each of the shard nodes based on the busyness of each of the shard nodes, and send the calculation result to each of the corresponding shard nodes; the collection time window consists of a reference time collection window and an overlap time collection window;

[0107] A hybrid log interception unit is used for each shard node to receive the corresponding collection time window, and through the backup agent, periodically intercept the operation log and the storage engine checkpoint log based on the respective collection time window, and upload them to the backup storage system.

[0108] Based on the above embodiment, the acquisition time window determining unit may include:

[0109] A collection subunit, configured for the backup agent of each shard node to periodically collect the write operation frequency and write data bandwidth of each shard and send the data to the central tuner;

[0110] The calculation subunit is used for the central tuner to obtain the write operation frequency and the write data bandwidth of all shards, calculate the acquisition time window of each shard node, and send it to the corresponding shard node.

[0111] Based on the above embodiment, the hybrid log interception unit may include:

[0112] The subunit for determining the starting point of this acquisition is configured to use the previous acquisition end point as a reference and shift the overlapping time acquisition window forward as the starting point of this acquisition; the previous acquisition end point is generated based on the triggering of the reference time acquisition window;

[0113] The interception subunit is used to intercept the operation log and the storage engine checkpoint log from the starting point of the current collection, and upload them to the backup storage system.

[0114] It should be noted that the order of the modules and units in the above data backup device can be changed without affecting the logic.

[0115] The data backup device provided by the embodiment of the present invention uses a full backup module 100 to coordinate the nodes in the cluster to perform a global logical time consistency backup using a central tuner when a full backup is triggered; the nodes include each shard node and a configuration server node; the incremental backup module 200 is used to intercept and back up the operation log and storage engine checkpoint log based on a time window overlapping algorithm using the backup agent of each shard node when an incremental backup is triggered. The beneficial effects of this device are: during a full backup, the central tuner tunes each shard in the cluster to generate consistent backup data, which ensures that the data of each shard can be restored to the same logical time point during data recovery; during an incremental backup, each backup agent collects a mixed log, and the risk of losing high-frequency instantaneous data in the operation log is eliminated through a window overlapping algorithm.

[0116] Figure 5 A schematic diagram of the structure of a data backup device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the data backup device includes:

[0117] Memory 60, for storing computer programs;

[0118] The processor 61 is configured to implement the steps of the data backup method of the above embodiment when executing a computer program.

[0119] The data backup device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0120] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing content required to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor for handling computational operations related to machine learning.

[0121] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps of the data backup method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include but is not limited to data of the data backup method, etc.

[0122] In some embodiments, the data backup device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0123] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation to the data backup device, and may include more or fewer components than shown in the figure.

[0124] It is understood that if the data backup method in the above-mentioned embodiment is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and performs all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM, a register, a hard drive, a removable disk, a CD-ROM, a magnetic disk, or an optical disk, and other media that can store program code.

[0125] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data backup method are implemented.

[0126] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below can be referenced with other embodiments described herein.

[0127] A computer program product includes a computer program / instruction, which implements the steps of the aforementioned disclosed data backup method when executed by a processor.

[0128] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0129] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0130] Finally, it should be noted that, in this document, relationships such as first and second, etc., are used solely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0131] The above is a detailed introduction to a data backup method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A data backup method, characterized in that: include: When a full backup is triggered, the central tuner is used to coordinate the nodes in the cluster to perform a global logical time consistency backup; The nodes include various shard nodes and configuration server nodes; When an incremental backup is triggered, the backup agent of each shard node is used to intercept and back up the operation log and the storage engine checkpoint log based on a time window overlapping algorithm.

2. The data backup method according to claim 1, wherein: Trigger a full backup, including: Using the central tuner to monitor changes in metadata on the configuration server node in real time; When it is detected that the cluster is scaled in or out or data is migrated, the full backup is triggered.

3. The data backup method according to claim 1, wherein: Use the central tuner to coordinate the global logical time consistency backup of each node in the cluster, including: Obtaining, through the central tuner, a current operation log timestamp of each node in the cluster, and selecting a global maximum timestamp from a plurality of the current operation log timestamps as a global checkpoint timestamp; Through the backup agent of each node, the operation log is intercepted according to the global checkpoint timestamp, and the storage engine interface is called to generate a storage engine consistency snapshot, and the storage engine consistency snapshot and the intercepted operation log are uploaded to the backup storage system.

4. The data backup method according to claim 3, wherein: Obtaining, through the central tuner, the current operation log timestamp of each node in the cluster, including: Sending a snapshot preparation instruction to all the nodes in the cluster through the central tuner; When each of the nodes receives the snapshot preparation instruction, it suspends the data shard migration operation, records the current operation log timestamp and sends it to the central tuner.

5. The data backup method according to any one of claims 1 to 4, characterized in that: Utilizing the backup agent of each shard node, based on the time window overlap algorithm, the operation log and the storage engine checkpoint log are intercepted and backed up, including: The central tuner calculates the collection time window of each shard node based on the busyness of each shard node and sends it to the corresponding shard node; the collection time window consists of a reference time collection window and an overlap time collection window; Each of the shard nodes receives the corresponding collection time window, and through the backup agent, periodically intercepts the operation log and the storage engine checkpoint log based on the respective collection time window, and uploads them to the backup storage system.

6. The data backup method according to claim 5, characterized in that: Calculating, by the central tuner, a collection time window for each shard node based on the busyness of each shard node, and sending the result to the corresponding shard node, including: The backup agent of each shard node periodically collects the write operation frequency and write data bandwidth of the respective shards and sends the data to the central tuner; The central tuner obtains the write operation frequency and the write data bandwidth of all shards, calculates the acquisition time window of each shard node, and sends it to the corresponding shard node.

7. The data backup method according to claim 5, characterized in that: Periodically intercepting the operation log and the storage engine checkpoint log based on the respective collection time windows and uploading them to the backup storage system includes: Taking the previous acquisition end point as a reference, the overlapping time acquisition window is shifted forward as the starting point of this acquisition; the previous acquisition end point is generated based on the trigger of the reference time acquisition window; The operation log and the storage engine checkpoint log are intercepted from the starting point of the current collection and uploaded to the backup storage system.

8. A data backup device, characterized in that: include: The full backup module is used to coordinate the nodes in the cluster to perform global logical time consistency backup using the central tuner when a full backup is triggered; The nodes include various shard nodes and configuration server nodes; The incremental backup module is used to intercept and back up the operation log and storage engine checkpoint log based on the time window overlapping algorithm by using the backup agent of each shard node when the incremental backup is triggered.

9. A data backup device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the data backup method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the data backup method according to any one of claims 1 to 7 is implemented.

Images