Distributed database backup method and device and medium

Abstract

The invention discloses a distributed database backup method and device and a medium, and relates to the field of databases, the method comprises the following steps: receiving a backup instruction of a target database, analyzing the backup instruction, extracting a backup type and a backup target range in the backup instruction, and generating a corresponding backup task; when the full backup task is executed, an initial logic clock is generated, and an initial blood relationship topological graph is constructed; creating a full-amount atomic snapshot, partitioning the full-amount atomic snapshot and storing the full-amount atomic snapshot into a corresponding physical backup medium to generate full-amount metadata; when the incremental backup task is executed, historical backup metadata is obtained, a change log in the target database is scanned according to a historical logic clock, and the historical blood relationship topological graph is updated based on a change record; determining a newly-added storage unit, creating a newly-added atomic snapshot, and storing the newly-added atomic snapshot in a physical backup medium; incremental metadata is generated. The integrity and accuracy of incremental data are ensured, and the risk of data omission or wrong backup is reduced.

Description

Technical Field

[0001] This application relates to the field of database technology, specifically to a distributed database backup method, device, and medium. Background Technology

[0002] With the advent of the big data era, data volume is growing explosively, and distributed databases are playing an increasingly important role in data storage and management, finding widespread application in large-scale data storage and real-time read / write scenarios. In practical applications, to ensure data security within the database, backup operations are necessary.

[0003] Database backups are a fundamental guarantee of data security, and are generally performed using full backups and incremental backups. Full backups require periodically copying the entire database dataset, while incremental backups achieve efficient incremental data capture by tracking file timestamp changes.

[0004] Traditional database backup methods typically use physical disk replication. When performing a full backup, the time and storage costs increase significantly because disk files are continuously merged and reconstructed during compression, making it difficult to meet the real-time protection needs of scenarios with high-frequency data updates. When performing incremental backups, due to the distributed architecture of distributed databases, the markers generated by deletion operations physically disappear after disk file compression. This makes it impossible to trace historical deletion records in incremental backups, and the compression and merging process disrupts the file timestamp sequence, causing the incremental backup chain to break and easily leading to data omissions or erroneous backups. Summary of the Invention

[0005] To address the aforementioned problems, this application proposes a distributed database backup method, comprising:

[0006] The system receives a backup instruction for the target database, parses the instruction, extracts the backup type and target range, and generates a corresponding backup task. The backup type includes full backup and incremental backup.

[0007] When performing a full backup task, query the backup management database to see if the full backup metadata corresponding to the target database exists, in order to determine whether it is the first full backup.

[0008] If it does not exist, then obtain the storage information corresponding to each storage unit within the backup target range, generate an initial logical clock, and construct the corresponding initial lineage topology map;

[0009] create a full-amount atomic snapshot corresponding to the storage unit, block the full-amount atomic snapshot and store it into the corresponding physical backup medium, generate full-amount metadata based on the initial logical clock, the initial blood relationship topology graph and the storage index, and store the full-amount metadata into the backup management database;

[0010] When the incremental backup task is executed, the historical backup metadata corresponding to the last backup in the backup management database is acquired, the change log in the target database is scanned according to the historical logical clock in the historical backup metadata, the change record is extracted, and the historical blood relationship topology graph is updated based on the change record;

[0011] According to the updated historical blood relationship topology graph, the new storage unit is determined, the new atomic snapshot corresponding to the new storage unit is created, the new atomic snapshot is blocked, and the new atomic snapshot is stored into the corresponding physical backup medium;

[0012] Generate incremental metadata based on the current logical clock, the updated blood relationship topology graph and the new storage index, and store the incremental metadata into the backup management database.

[0013] On the other hand, the application also proposes a distributed database backup device, comprising:

[0014] at least one processor; and,

[0015] a memory in communication connection with the at least one processor; wherein,

[0016] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a distributed database backup method as described in the above examples.

[0017] On the other hand, the application also proposes a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are set as a distributed database backup method as described in the above examples.

[0018] The distributed database backup method proposed by the application can bring the following beneficial effects:

[0019] The application determines whether it is the first full-amount backup by querying the backup management database, generates the initial logical clock and blood relationship topology graph when it is the first full-amount backup, and accurately captures the initial state of the storage unit. If it is not the first full-amount backup, the current active storage unit is filtered based on the historical full-amount metadata, invalid processing of retired or redundant data is avoided, the efficiency of full-amount backup is significantly improved, and the full-amount backup can better adapt to the backup requirements in the high-frequency data update scenario.

[0020] In the incremental backup link, the logical clock anchor time starting point in the historical backup metadata is combined with the change log to extract complete change records, especially through the dynamic update and association index construction of the blood relationship topology graph, the effective tracing of the historical deletion operation is realized. Breakthrough the limitation of traditional incremental backup due to disk file compression, deletion mark disappearance and timestamp sequence destruction, avoid the problem of incremental backup chain rupture, ensure the integrity and accuracy of incremental data, reduce the risk of data omission or error backup. BRIEF DESCRIPTION OF DRAWINGS

[0021] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 A flowchart of a distributed database backup method in an embodiment of the present application is shown.

[0023] Figure 2 A schematic diagram of a distributed database backup device in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme of the present application will be described in detail below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0025] The technical scheme provided by each embodiment of the present application will be described in detail below in combination with the drawings.

[0026] As shown in Figure 1 The present application provides a distributed database backup method, comprising:

[0027] S101: receiving a backup instruction of a target database, analyzing the backup instruction, extracting the backup type and backup target range in the backup instruction, and generating a corresponding backup task; the backup type includes full backup and incremental backup.

[0028] The database management interface (such as API, command line tool or management platform) receives the backup instruction triggered by the external, and the instruction source can include timing task, manual trigger or abnormal recovery trigger.

[0029] The instruction content is parsed, the backup type and the backup target range are extracted, the backup type includes full backup and incremental backup, the backup target range is a specific object to be backed up, such as a whole cluster of a database, a specified key space, a specified table, shard data of a specific node, etc., and is converted into a range identifier recognizable by the database, such as a node ID list, a shard hash interval, a table name set, etc.

[0030] The parsing result is encapsulated as a structured task object, including task ID, target database identifier, backup type, target range, priority, timeout threshold and other information, and is stored in a task queue for scheduling.

[0031] S102: When a full backup task is executed, it is queried whether the full backup metadata corresponding to the target database exists in the backup management database, to determine whether it is the first full backup.

[0032] When the backup type is full backup, it is queried whether the full backup metadata corresponding to the target database exists in the backup management database, to determine whether it is the first full backup.

[0033] The backup management database is a structured storage specially storing backup metadata, and can be queried whether there is historical full backup metadata by taking the target database identifier and the full backup type as query conditions.

[0034] S103: If not, the storage information of each storage unit in the backup target range is obtained, an initial logical clock is generated, and a corresponding initial blood relation topology graph is constructed.

[0035] If the query result is empty, or the historical full metadata has been marked as "expired" or "invalid", it is determined that it is the first full backup.

[0036] The storage information of the storage unit is obtained, all nodes in the backup target range are traversed, the underlying storage unit is scanned through a storage engine interface, and the core storage metadata of each storage unit is collected, including a unique identifier, a primary key range, a creation timestamp, a source data block identifier set, a file size, a checksum value and a storage unit address.

[0037] The maximum timestamp of all storage units is aggregated, the maximum value thereof is taken as a basic timestamp, the latest position identifier of the commit log in the backup target range is queried, the basic timestamp is associated and bound with the commit log position identifier, and a composite initial logical clock is generated.

[0038] The validity of the initial logical clock is verified, it is checked that the timestamps of all storage units are not greater than the basic timestamp, and the commit log position identifier is consistent with the current state of the cluster, if the verification is passed, it is confirmed that the initial logical clock is valid, otherwise the step of generating the initial logical clock is re-executed.

[0039] Based on the storage information of each storage unit in the backup target range, a topology graph node is created for each storage unit, which contains a unique identifier of the storage unit, a creation timestamp, a data range (a start value and an end value of a primary key), a physical node identifier to which the storage unit belongs, and a set of source data block identifiers.

[0040] The set of source data block identifiers of each storage unit is parsed to extract the unique identifiers of the upstream data blocks recorded therein, a mapping relationship between the current storage unit and the upstream data blocks is established, a directed edge of the topology graph is constructed according to the mapping relationship, the topology graph nodes are connected through the directed edge, and a bloodline topology graph is constructed.

[0041] If the set of source data block identifiers of the storage unit is empty, the node is a root node of the topology graph, and no incoming edge is created. If the storage unit is directly derived from a single upstream data block, a one-way edge is created from the upstream data block node to the current storage unit node, and the edge attribute is marked as "directly derived". If the storage unit is generated by merging multiple upstream data blocks, one-way edges are created from each upstream data block node to the current storage unit node, the edge attribute is marked as "merge derived", and the timestamp of the merge operation is recorded.

[0042] The integrity of the topology graph is checked to ensure that all non-root nodes have at least one incoming edge to ensure traceability of the bloodline, and the creation timestamps of all nodes are not later than the initial logical clock to ensure consistency with the reference time.

[0043] After the verification, the initial bloodline topology graph and the initial logical clock are stored in association, and a unique version identifier is assigned to the topology graph as a reference version for subsequent topology graph updates.

[0044] In addition, if there is full backup metadata corresponding to the target database, it is determined that it is not the first full backup.

[0045] The historical bloodline topology graph, the historical logical clock, and the historical storage index in the historical full metadata are extracted. The historical bloodline topology graph is used to record the bloodline relationship of the storage unit at the time of the last full backup; the historical logical clock is used as the time reference of the last full backup; and the historical storage index is used to record the storage location of the data block of the last full backup.

[0046] All nodes in the backup target range in the historical bloodline topology graph are traversed, the storage units in the current physical storage are scanned, the non-active units are excluded in combination with the elimination markers in the historical bloodline topology graph, and the current active storage units are screened. Among them, the non-active units are the storage units that have been marked as merged and eliminated in the historical topology graph (such as units A and B have been merged into C, and A and B are no longer active data), or the redundant storage units whose data range is completely covered by new units.

[0047] The steps of generating the initial logical clock and constructing the blood relationship topology graph are performed to generate the latest logical clock and construct the latest blood relationship topology graph to establish the time reference and storage unit blood relationship of the current full backup, replacing the historical reference.

[0048] In the embodiments of the present application, data is read from the nodes in the determined backup range. For each node, data in each storage unit (SSTable file) in each backup target range is sequentially read according to the data storage structure of the distributed database. For example, by traversing the data directory of the node, the corresponding SSTable file is found, and the data in the file is read into the memory using the underlying data reading interface or tool provided by the Cassandra database.

[0049] S104: Create a full atomic snapshot corresponding to the storage unit, block the full atomic snapshot and store it in the corresponding physical backup medium, generate full metadata based on the initial logical clock, the initial blood relationship topology graph and the storage index, and store it in the backup management database.

[0050] When performing the first full backup, a snapshot locking instruction is sent to all nodes in the target range, so that the storage unit enters a read-only state, ensuring that the data is not modified concurrently during the snapshot.

[0051] A physical copy of the read-only storage unit is created through file system hard link or storage engine snapshot interface to generate a timestamp-unique full atomic snapshot. The snapshot is immediately unlocked after generation, restoring the node's read-write capability and avoiding long-term blocking of the cluster.

[0052] Read the snapshot file, block it according to the preset rules, get the corresponding block data, generate a block unique identifier and a check value for each block of data, upload the block data to the physical backup medium (such as local disk array, distributed storage HDFS, cloud storage S3), and record the storage path, size and check value of each block to form a corresponding block-level storage index.

[0053] Based on the initial logical clock, the initial blood relationship topology graph and the block-level storage index, full metadata is generated and stored in the backup management database.

[0054] When performing a non-first full backup, an active atomic snapshot corresponding to the current active storage unit is created, the active atomic snapshot is converted into a universal format, and is stored in the corresponding physical backup medium in blocks to generate a corresponding current block storage index.

[0055] Based on the latest logical clock, the latest blood relationship topology graph and the current block storage index, the latest full metadata is generated and stored in the backup management database.

[0056] In the embodiments of the present application, necessary conversion and arrangement are performed on the data. For example, the Cassandra-specific data format is converted into a general backup data format to facilitate subsequent storage and recovery operations. At the same time, checksums are calculated for the data to generate checksum values for each data block or record, which are used to verify the integrity of the data when the data is recovered. The converted and arranged data is stored in a full backup file. Depending on the backup range and the size of the data amount, a suitable storage method can be selected. For example, for a smaller data amount, all data can be stored in one file; for a larger data amount, the data can be stored in multiple files in blocks, and corresponding index files can be established to facilitate quick positioning and recovery of the data. The full backup file can be stored in a local disk, a network shared storage, or a cloud storage, etc.

[0057] In the embodiments of the present application, after the memory data is refreshed and written to the disk, the time point of this full backup is recorded. The time point information can be stored in a metadata file associated with the full backup file, or in a special backup management database. The time point information will serve as an important reference basis for subsequent incremental backup and data recovery.

[0058] S105: When performing an incremental backup task, the historical backup metadata corresponding to the last backup in the backup management database is obtained, the change log in the target database is scanned according to the historical logical clock in the historical backup metadata, the change record is extracted, and the historical blood relationship topology graph is updated based on the change record.

[0059] When performing an incremental backup task, the historical backup metadata corresponding to the last backup in the backup management database is obtained, and it is judged whether the backup identification corresponding to the historical backup metadata is an incremental backup identification, to determine whether it is the first incremental backup.

[0060] If not, it is determined that the historical backup metadata is historical full metadata, and it is the first incremental backup.

[0061] The historical logical clock in the historical full metadata is set as a first time starting point, the first change log after the first time starting point in the target database is scanned, all data changes occurring after the full backup are extracted, including the first deletion operation record and the first addition operation record.

[0062] Among them, the deletion operation record contains the deleted key-value pair, the deletion timestamp, the storage unit ID, etc.; the addition operation record contains the newly created storage unit ID, the creation timestamp, the data range, the source data block, etc.

[0063] Structuring the unique identifier of the deleted data in the first deletion operation record, the timestamp of the deletion operation, and the storage location of the data in the full backup, a first incremental deletion set is generated, and a first association index of the first incremental deletion set and the target database is constructed. The first association index includes: associating the deleted data with the ID of the corresponding data block in the full backup, and associating the difference between the deletion timestamp and the full logical clock.

[0064] Based on the first addition operation record and the first association index, the historical blood lineage topology graph in the historical full metadata is updated.

[0065] The new storage unit information in the first addition operation record is parsed, and the attributes of each new unit are determined. If it is a newly generated storage unit, it is marked as a root node candidate; if it is a merged generated storage unit, the source unit ID and the merging timestamp are recorded.

[0066] The storage unit affected by the deletion operation is queried through the first association index. If a newly added storage unit contains deleted data, mark the part of the data as invalid in the topology graph, and only keep the valid data range that has not been deleted.

[0067] Based on the above addition processing and deletion processing, the blood lineage topology graph is updated. The new storage unit in the first addition operation record is added to the topology graph as a node, and its unique identifier, creation timestamp, and valid data range are written. If the new unit is generated by merging units in the full backup, a directed edge from the full unit node to the new unit node is added, and the edge attribute is marked as merging derivation. If it is newly generated, no edge is added. Mark the obsolete nodes: if some units in the full backup have been merged into new units, mark these old units as obsolete in the topology graph, and they are no longer active nodes.

[0068] If yes, it is determined that the historical backup metadata is historical incremental metadata, which is not the first incremental backup.

[0069] The historical logical clock in the historical incremental metadata is set as the second time starting point, the second change log after the second time starting point in the target database is scanned, and the second deletion operation record and the second addition operation record are extracted.

[0070] According to the historical blood lineage topology graph in the historical incremental metadata, the antecedent deletion set associated with the second deletion operation record is determined.

[0071] Based on the second deletion operation record, a non-first incremental deletion set is generated, and a second association index of the non-first incremental deletion set and the antecedent deletion set is constructed.

[0072] Based on the second addition operation record and the second association index, the historical blood lineage topology graph in the historical incremental metadata is updated.

[0073] S106: According to the updated historical blood relationship topology map, determine the newly added storage unit, create the corresponding newly added atomic snapshot, block the newly added atomic snapshot, and store it in the corresponding physical backup medium.

[0074] When it is the first incremental backup, filter the nodes in the updated blood relationship topology map that are not marked as obsolete, have no successor nodes, and have a creation timestamp greater than the historical logical clock, generate a set of newly added storage units, and target data for incremental backup.

[0075] Only the snapshot operation is performed on the set of newly added storage units, without locking the entire cluster, only locking the target unit, generating a snapshot copy through hard link, and ensuring data consistency.

[0076] If the new unit contains modifications to historical data, the modification range needs to be marked separately to avoid duplicating unchanged parts.

[0077] The new snapshot is blocked, an incremental storage index is generated, and if the new data block is repeated with the historical backup block, the historical checksum can be reused, only the difference is stored, and redundancy is reduced.

[0078] When it is not the first incremental backup, the last logical clock, the last blood relationship topology map, the last incremental backup recorded deletion operation, and the last storage index are obtained from the last incremental metadata.

[0079] Set the last logical clock as the time boundary, only scan the change log after that time point, extract the second change log, the change log contains the creation, merge, and obsolescence of the storage unit, as well as the addition and deletion of data events, and the records with timestamp and commit log position greater than the last logical clock need to be filtered.

[0080] The second change log includes the second deletion operation record and the second addition operation record.

[0081] The second deletion operation record is structured and processed to generate a non-first incremental deletion set, including the unique identifier of the deleted data, the timestamp of the deletion operation, the source marker of the deleted data, etc., and the association between the current non-first incremental deletion set and the last incremental deletion set is established.

[0082] Parse the source of the new storage unit, if the new unit is generated by merging the active unit in the last incremental, record the source unit ID merge timestamp, if it is newly generated, mark it as a new root node.

[0083] Filter invalid units by combining chain association index, identify the previous incremental units affected by the current deletion set through index, mark their valid data range in the topology map, and only keep the un-deleted part.

[0084] Add the new storage unit in the second incremental operation record to the topology graph, write its identifier, creation timestamp, valid data range, etc. If the new unit is derived from the unit of the last increment, add a directed edge from the historical unit node to the new unit node, and mark the edge attribute as incremental derivation. For the units in the last incremental topology graph that have been merged and eliminated or completely covered by the current deletion set, mark them as invalid and no longer as active nodes.

[0085] In the updated topology graph, filter the nodes with the following conditions: active and not marked as invalid, no successor node, creation timestamp greater than the last logical clock, and valid data range not covered by the current deletion set.

[0086] Only perform snapshots on the filtered new storage units, without locking the entire cluster, only locking the target unit, ensuring data consistency.

[0087] Compare the new unit data blocks with the last incremental storage index and the full storage index. If there are duplicate data blocks, directly reuse the historical storage location.

[0088] Divide the new data blocks into blocks, store them to the corresponding physical backup medium, and generate a non-first incremental storage index to record the storage location of the new blocks and the association relationship of the reused blocks.

[0089] S107: Based on the current logical clock, update the bloodline topology graph and the new storage index to generate incremental metadata and store it in the backup management database.

[0090] When it is the first incremental backup, generate the first incremental metadata based on the task ID, current logical clock, updated bloodline topology graph, incremental storage index, associated parent backup identifier corresponding to the metadata ID pointing to the historical baseline, and deletion operation record, and store it in the backup management database.

[0091] When it is a non-first incremental backup, generate the incremental metadata based on the current logical clock, updated bloodline topology graph, non-first incremental storage index, non-first incremental deletion set and chain association index, associated parent backup identifier corresponding to the metadata ID of the last incremental backup, and complete incremental chain identifier, and store it in the backup management database for sequential application during recovery.

[0092] In the embodiments of the present application, for incremental backup, according to the time point of full backup and the backup strategy set by the user, the range of incremental backup (consistent with the range of full backup or a subset thereof) and the time interval are determined. For example, the user can set to perform incremental backup every hour, and the range of incremental backup is the same as the range of the last full backup.

[0093] Based on timestamp comparison, each SSTable file in the data storage structure of Cassandra has a corresponding creation timestamp. By comparing the current time with the time point of the last full backup or incremental backup, the newly created or updated SSTable files in the time interval are determined. For example, traverse the data directory of the node to obtain the timestamp information of each SSTable file, and filter out the files with timestamps within the backup time interval.

[0094] Using change logs, in addition to timestamp comparison, the change logs (such as CommitLog) of Cassandra can also be used to more accurately detect data changes. Analyze the operations recorded in the change logs to determine the data involved in the insert, update, and delete operations that occurred within the backup time interval. By parsing the change log files, extract the relevant operation information and data records.

[0095] Creating incremental backup files, for the detected data changes, read the changed data from the corresponding SSTable files or according to the records in the change logs. For example, if it is a new SSTable file determined by timestamp comparison, read all the data in the file; if it is a changed data determined based on the change log, read the specific changed data from the corresponding SSTable file according to the record in the log.

[0096] Data processing and storage, similar data conversion and arrangement operations are performed on the read changed data as full backup, and then stored in the incremental backup file. The incremental backup file can also be stored in local disk, network shared storage or cloud storage, etc. storage medium. Similar to the full backup file, the appropriate storage method can be selected according to the size of the data, if the data is small, it can be stored in one file, and if it is large, it can be stored in blocks and indexed.

[0097] Marking the incremental backup time point, after completing the creation of the incremental backup file, record the time point of this incremental backup. Similarly, store the time point information in the metadata file associated with the incremental backup file or in the backup management database.

[0098] The present application determines whether it is the first full backup by querying the backup management database, generates an initial logical clock and bloodline topology graph when it is the first full backup, and accurately captures the initial state of the storage unit; if it is not the first full backup, filter the current active storage unit based on the historical full metadata, avoid invalid processing of obsolete or redundant data, significantly improve the efficiency of full backup, and make it better adapt to the backup needs in high-frequency data update scenarios.

[0099] In the incremental backup link, the logical clock anchor time starting point in the historical backup metadata is combined with the change log to extract complete change records, especially through the dynamic update and association index construction of the blood relationship topology graph, the effective tracing of the historical deletion operation is realized. Break through the limitation of the disappearance of the deletion mark and the destruction of the timestamp sequence caused by the disk file compression in the traditional incremental backup, avoid the problem of the breakage of the incremental backup chain, ensure the integrity and accuracy of the incremental data, reduce the risk of data omission or error backup.

[0100] As Figure 2 shown, the embodiment of the present application also provides a distributed database backup device, comprising:

[0101] at least one processor; and,

[0102] a memory in communication with the at least one processor; wherein,

[0103] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the distributed database backup method according to any one of the above embodiments.

[0104] The embodiment of the present application also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to perform the distributed database backup method according to any one of the above embodiments.

[0105] Each embodiment in the present application adopts a progressive manner for description, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0106] The device and medium provided by the embodiment of the present application are one-to-one corresponding with the method, so the device and medium also have the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be described here.

[0107] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0108] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0109] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0110] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0111] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0112] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer-readable media.

[0113] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0114] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0115] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method of distributed database backup, characterized by, The method comprises the following steps: receiving a backup instruction of a target database, parsing the backup instruction, extracting the backup type and backup target range in the backup instruction, and generating a corresponding backup task; the backup type includes full backup and incremental backup; when performing a full backup task, querying whether there is full backup metadata corresponding to the target database in the backup management database to determine whether it is the first full backup; if not, obtaining the storage information of each storage unit in the backup target range, generating an initial logical clock, and constructing a corresponding initial blood relationship topology graph; creating a full atomic snapshot corresponding to the storage unit, storing the full atomic snapshot into the corresponding physical backup medium after being divided into blocks, generating full metadata based on the initial logical clock, the initial blood relationship topology graph, and the storage index, and storing the full metadata into the backup management database; when performing an incremental backup task, obtaining historical backup metadata corresponding to the last backup in the backup management database, scanning the change log in the target database according to the historical logical clock in the historical backup metadata, extracting change records, and updating the historical blood relationship topology graph based on the change records; determining the newly added storage unit according to the updated historical blood relationship topology graph, creating a newly added atomic snapshot corresponding to the newly added storage unit, and storing the newly added atomic snapshot into the corresponding physical backup medium after being divided into blocks; generating incremental metadata based on the current logical clock, the updated blood relationship topology graph, and the newly added storage index, and storing the incremental metadata into the backup management database.

2. The method of claim 1, wherein, After querying whether there is full backup metadata corresponding to the target database in the backup management database to determine whether it is the first full backup, the method further comprises: if there is, obtaining the historical full metadata corresponding to the last full backup in the backup management database based on the full backup identifier, and according to the historical blood relationship topology graph in the historical full metadata, the current active storage unit in the backup target range; generating the latest logical clock based on the current storage information of the current active storage unit in the target database, and constructing the latest blood relationship topology graph corresponding to the current active storage unit; creating an active atomic snapshot corresponding to the current active storage unit, converting the active atomic snapshot into a universal format, storing the active atomic snapshot into the corresponding physical backup medium after being divided into blocks, and generating a corresponding current block storage index; generating the latest full metadata based on the latest logical clock, the latest blood relationship topology graph, and the current block storage index, and storing the latest full metadata into the backup management database.

3. The method of claim 1, wherein, According to the historical logical clock in the historical backup metadata, the method further comprises the following steps: determining whether the backup identifier corresponding to the historical backup metadata is an incremental backup identifier to determine whether it is the first incremental backup; if not, the historical backup metadata is historical full metadata. Set a historical logical clock in the historical full amount metadata as a first time starting point, scan a first change log after the first time starting point in the target database, and extract a first deletion operation record and a first addition operation record; Based on the first deletion operation record, generate a first incremental deletion set, and construct a first association index of the first incremental deletion set and the target database; Based on the first addition operation record and the first association index, update a historical blood relationship topology graph in the historical full amount metadata.

4. The method of claim 3, wherein, After determining whether the backup identification corresponding to the historical backup metadata is an incremental backup identification, to determine whether it is after a first incremental backup, the method further includes: If not, determine that the historical backup metadata is historical incremental metadata; Set a historical logical clock in the historical incremental metadata as a second time starting point, scan a second change log after the second time starting point in the target database, and extract a second deletion operation record and a second addition operation record; According to a historical blood relationship topology graph in the historical incremental metadata, determine a previous deletion set associated with the second deletion operation record; Based on the second deletion operation record, generate a non-first incremental deletion set, and construct a second association index of the non-first incremental deletion set and the previous deletion set; Based on the second addition operation record and the second association index, update the historical blood relationship topology graph in the historical incremental metadata.

5. The method of claim 1, wherein, The obtaining of the storage information corresponding to each storage unit in the backup target range and the generation of an initial logical clock specifically include: Obtain the storage information corresponding to each storage unit in the backup target range; the storage information includes a storage unit unique identifier, a creation timestamp, a source data block identifier set, and a storage unit address; Compare the creation timestamps corresponding to the storage units to determine a maximum value, and set the creation timestamp corresponding to the maximum value as a basic timestamp; Determine a latest commit log address corresponding to the backup target range, associate the basic timestamp with the latest commit log address, and generate an initial logical clock corresponding to the backup target range.

6. The method of claim 5, wherein, The construction of the initial blood relationship topology graph corresponding to the storage unit specifically includes: Based on the storage information of each storage unit in the backup target range, create a topology graph node corresponding to each storage unit; Parse the source data block identifier set to extract an upstream data block identifier corresponding to the storage unit, and based on the upstream data block identifier, establish a mapping relationship between the storage unit and an upstream data block; According to the mapping relationship, construct a topology graph directed edge, associate the topology graph nodes, construct an initial blood relationship topology graph corresponding to the backup target range, and associate the initial blood relationship topology graph with the initial logical clock.

7. The method of claim 6, wherein, The construction of the topology graph directed edge according to the mapping relationship specifically includes: If the source data block identifier set of the current storage unit is empty, determine that the topology graph node corresponding to the current storage unit is a root node; If the source data block identifier set of the current storage unit contains a single identifier, determine the upstream data block node corresponding to the single identifier, and create a unidirectional edge from the upstream data block node to the node corresponding to the current storage unit; If the source data block identifier set of the current storage unit contains multiple identifiers, determine multiple upstream data block nodes corresponding to the multiple identifiers, and create unidirectional edges from the multiple upstream data block nodes to the node corresponding to the current storage unit.

8. The method of claim 1, wherein, The creating of the full-amount atomic snapshot corresponding to the storage unit, the blocking of the full-amount atomic snapshot, and the storage of the full-amount atomic snapshot into the corresponding physical backup medium, specifically include: The creating of the full-amount atomic snapshot corresponding to the storage unit, the conversion of the full-amount atomic snapshot into a general format, and the blocking of the full-amount atomic snapshot to obtain the blocked atomic data; The storage of the blocked atomic data into the corresponding physical backup medium, and the generation of the corresponding block-level storage index.

9. A distributed database backup device, characterized by Comprise: At least one processor; And The memory connected in communication with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the distributed database backup method according to any one of claims 1-8.

10. A non-transitory computer storage medium storing computer-executable instructions, the computer-executable instructions comprising instructions for: receiving a request to access a file; determining whether the file is stored in a cache; and in response to determining that the file is stored in the cache, providing access to the file from the cache. The computer executable instructions are configured to perform the distributed database backup method according to any one of claims 1-8.

Images