Method, device, equipment and medium for continuously maintaining final consistency of backup data

Abstract

The embodiment of the present invention provides a method, device, equipment and medium for continuously maintaining the final consistency of backup data, which relates to the field of distributed database technology, including that the data source side of the streaming backup is the backup source end, and the storage side of the backup data is the backup target end; in the distributed database cluster of the backup source end, a global consistency point is created; in the distributed database cluster of the backup target end, each data shard continuously receives a pre-written log stream and continuously saves it to persistent storage, and ensures that the log is stopped and played at the global consistency point through a preset control mechanism; when the backup target end is restored to a normal distributed database cluster, the invalid log is cleared and played to the nearest global consistency point to achieve data final consistency. The solution continuously maintains the final consistency of the backup data on the backup data side, effectively improving the speed of restoring to a readable and writable distributed database cluster.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed databases, and in particular to a method, device, equipment and medium for continuously maintaining the final consistency of backup data. Background Art

[0002] Data storage and real-time backup In modern databases, data is usually stored in two ways: write-ahead log files (for fast writing) and data content files (for querying). Write-ahead log files are usually I / O sequential write operations, which are faster to write. Data content files are usually I / O random write operations, which are slower to write.

[0003] Generally speaking, in order to balance the write performance, availability, and correctness of data storage, modern databases will first save data changes in a write-ahead log file when storing data, and then write the data itself into the data content file. If the data content file is damaged, it can be "played" from the write-ahead log file to regenerate the correct data content file.

[0004] Based on the above design, modern databases usually have two ways to back up: streaming backup of pre-written log files and direct backup of data content files. Generally speaking, the streaming backup of pre-written log files ensures a certain degree of real-time performance; while the direct backup of data content files generally cannot ensure real-time performance due to the limitation of data volume. When using the streaming backup of pre-written log files for backup, in order to maintain the availability of the backup data, the pre-written log file content is usually "played" as a new data content file on the backup side while streaming the pre-written log files. This allows the data to be quickly used for query on the backup side.

[0005] Regarding the consistency of distributed databases, in modern distributed storage distributed databases, data is usually stored in the form of multiple data shards in each node of the distributed database. At the same time, there is usually a data shard used to store the metadata of the data shard (data sharding method and strategy, etc.), and the node that stores this shard is generally used to coordinate data operations of other nodes.

[0006] Transactional distributed databases are usually able to coordinate data operations on each node on the distributed database cluster side through various mechanisms (various concurrency control mechanisms such as locks) to ensure transaction-level data consistency.

[0007] For real-time backup of distributed databases, generally speaking, each node of a distributed database will save data in both write-ahead log files and data content files, just like a stand-alone database. The backup of a distributed database is the backup of the write-ahead log files and data content files of multiple nodes.

[0008] However, the complexity of concurrent transaction processing in distributed databases makes it time-consuming to find a location where data is consistent in real time from backups of multiple data shards. This makes it very slow to restore a new available distributed database cluster from backup data. If these backup data are used for disaster recovery, it will directly affect one of the key indicators of disaster recovery: Recovery Time Object (RTO). Summary of the invention

[0009] In view of this, an embodiment of the present invention provides a method for continuously maintaining the final consistency of backup data to solve the technical problem of time-consuming operation of restoring a new available distributed database cluster from backup data when concurrently processing transactions of a distributed database in the prior art. The method includes:

[0010] Distributed databases use a write-ahead log streaming backup method to back up data. The data source side of the streaming backup is set as the backup source side, and the storage side of the backup data is set as the backup target side.

[0011] In the distributed database cluster of the backup source, a global consistency point is created according to a preset time interval, wherein the global consistency point is information stored in a write-ahead log;

[0012] In the distributed database cluster of the backup target end, each data shard continuously receives the pre-written log stream and continuously saves it to persistent storage, and ensures that the log is stopped and played at the global consistency point through a preset control mechanism;

[0013] When the backup target is restored to a common distributed database cluster, invalid logs are cleared and played to the nearest global consistency point to achieve final data consistency.

[0014] The embodiment of the present invention also provides a device for continuously maintaining the final consistency of backup data, so as to solve the technical problem of time-consuming operation of restoring a new available distributed database cluster from backup data when concurrently processing transactions of a distributed database in the prior art. The device includes:

[0015] A setting module is used for distributed databases to perform data backup using a pre-written log streaming backup method, setting the data source side of the streaming backup as the backup source end, and setting the storage side of the backup data as the backup target end;

[0016] A global consistency point creation module, used to create a global consistency point in the distributed database cluster of the backup source according to a preset time interval, wherein the global consistency point is information stored in a write-ahead log;

[0017] A backup module, which is used for each data shard to continuously receive the pre-written log stream in the distributed database cluster of the backup target end, and continuously save it to persistent storage, and ensure that the log is stopped and played at the global consistency point through a preset control mechanism;

[0018] The recovery module is used to clean up invalid logs and play to the nearest global consistency point to achieve final data consistency when the backup target end is restored to a common distributed database cluster.

[0019] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned methods for continuously maintaining the final consistency of backup data when executing the computer program, so as to solve the technical problem of time-consuming operation of restoring a new available distributed database cluster from backup data when concurrently processing transactions of a distributed database in the prior art.

[0020] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned methods for continuously maintaining the final consistency of backup data, so as to solve the technical problem of time-consuming operation of restoring a new available distributed database cluster from backup data when concurrently processing transactions of a distributed database in the prior art.

[0021] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: the present application adopts a method of periodically creating global consistency points at the source end of the distributed database backup, and controlling the data synchronization progress of each data shard based on the information of these global consistency points at the backup end. During recovery, by clearing invalid logs and accurately controlling log playback, it is ensured that the data is restored to a state consistent with the main database, and that the backup data can continue to maintain the ultimate consistency of the data content in the distributed database, thereby effectively improving the speed of recovery to a readable and writable distributed database cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 is a flow chart of a method for continuously maintaining final consistency of backup data provided by an embodiment of the present invention;

[0024] Figure 2is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0025] Figure 3 It is a structural block diagram of a device for continuously maintaining the final consistency of backup data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0027] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0028] In an embodiment of the present invention, a method for continuously maintaining the final consistency of backup data is provided, such as Figure 1 As shown, the method includes:

[0029] Step S101: The distributed database uses a write-ahead log streaming backup method to perform data backup, setting the data source side of the streaming backup as the backup source end, and setting the storage side of the backup data as the backup target end;

[0030] Step S102: in the distributed database cluster of the backup source, a global consistency point is created according to a preset time interval, wherein the global consistency point is information stored in a write-ahead log;

[0031] Step S103: In the distributed database cluster of the backup target end, each data shard continuously receives the write-ahead log stream and continuously saves it to persistent storage, and ensures that the log is stopped and played at the global consistency point through a preset control mechanism;

[0032] Step S104: When the backup target end is restored to a common distributed database cluster, invalid logs are cleared and played to the nearest global consistency point to achieve final data consistency.

[0033] In this embodiment, a method is adopted to periodically create global consistency points at the source end of the distributed database backup, and control the data synchronization progress of each data shard based on the information of these global consistency points at the backup end. During recovery, by clearing invalid logs and accurately controlling log playback, it is ensured that the data is restored to a state consistent with the main database, and that the backup data can continue to maintain the ultimate consistency of the data content in the distributed database, thereby effectively improving the speed of recovery to a readable and writable distributed database cluster.

[0034] In specific implementation, the global consistency point is a kind of information stored in the write-ahead log, and its existence form is different in different data shards.

[0035] In a normal data shard, a global consistency point is just a special write-ahead log record that marks the position in the write-ahead log for this shard.

[0036] In the shard that stores metadata, the global consistency point is also a special write-ahead log record, but this record stores the location information of the write-ahead log records of this point in all normal data shards. This record also marks the location of this point in the write-ahead log of the metadata shard.

[0037] In one embodiment, a distributed database cluster at a backup source end may periodically create a global consistency point, and the time interval for creation may be configured by a user. The creation of a global consistency point includes:

[0038] Acquire the global distributed write lock;

[0039] Write a pre-written log record of the global consistency point in all common data shard nodes of the distributed database, and send the location of this record to the node where the metadata shard is located;

[0040] Write the pre-written log record of the global consistency point to the node where the metadata shard is located, and save the location information and the shard to which the pre-written log records in all the common data shard nodes are obtained;

[0041] Release the global distributed write lock.

[0042] In specific implementation, a global distributed write lock is acquired, which ensures that there are no ongoing write transactions in the distributed database at this time. This ensures that when writing to the write-ahead log, the entry at this global consistency point is the latest record in all shards. The global consistency point ensures that when all data shards "play" the write-ahead log to this point, the data content file of the distributed database is in a final consistent state.

[0043] In one embodiment, the step of ensuring that the log is stopped and played at the global consistency point by a preset control mechanism includes:

[0044] On the node of the common data shard of the backup target end, the write-ahead log is continuously played as a data content file through a first preset control mechanism;

[0045] On the node of the metadata shard of the backup target end, the second preset control mechanism is used to continuously play the write-ahead log as a data content file, save the specified write-ahead log location information to persistent storage, and control the playback of the write-ahead log of the ordinary data shard of the backup target end.

[0046] In specific implementation, when backing up data, the distributed database cluster at the backup target end cannot read or write each data shard and continuously receives the pre-written log stream, and continuously saves it to persistent storage, and ensures that the log is stopped and played at the global consistency point through a preset control mechanism.

[0047] In one embodiment, the first preset control mechanism includes a first sub-mechanism, a second sub-mechanism and a third sub-mechanism.

[0048] The first sub-mechanism is configured to stop the playback behavior before saving the write-ahead log at the specified location to the persistent storage;

[0049] The second sub-mechanism is configured to save the specified write-ahead log location information to persistent storage;

[0050] The third sub-mechanism is configured to start the playback behavior and stop the playback behavior after the write-ahead log at the specified position is played as a data content file.

[0051] For the first sub-mechanism, the process of converting log records into data content files is stopped before a certain specified position in the write-ahead log. It has the following advantages: Preventing data corruption: ensuring that incomplete or inconsistent log records are not converted into data files when processing global consistency points, thereby avoiding data corruption; Precise control: allowing the system to pause log playback at a specific point in time to perform other operations, such as parsing global consistency point information; Resource management: when log playback needs to be paused, system resources can be released for other tasks.

[0052] Setting up the second sub-mechanism has the following advantages: Recording status: saving the location information of the currently processed write-ahead log so that the system knows where to continue processing the log when it is restarted or restored; Consistency check: during the recovery process, the location information can be used to verify the consistency of the data to ensure that the recovered data is consistent with the primary database; Fault recovery: in the event of a fault, the last processed location can be quickly located to reduce recovery time and complexity.

[0053] Setting up the third sub-mechanism has the following advantages: Precise playback: ensuring that only necessary log records are converted into data files when processing global consistency points to avoid unnecessary processing; Synchronizing data: ensuring that the data content in the data shards at the backup target end is consistent with the data in the primary database at the global consistency point; Resource optimization: by precisely controlling the playback range, the use of system resources can be optimized and the efficiency of backup and recovery can be improved.

[0054] The above sub-mechanisms work together during the backup process to ensure the final consistency of the backup data.

[0055] In one embodiment, the second preset control mechanism includes a loop of the following steps, and the loop is set at the node where the metadata shard is located:

[0056] Stop the playback of write-ahead logs;

[0057] Stream and parse the write-ahead log content from the stop playback position until the next global consistency point content is found and extracted;

[0058] Parse the content of the global consistency point record and obtain the location of each record of the global consistency point in the write-ahead log of all common data shards;

[0059] Use the function of the first sub-mechanism to control all common data shards to stop playing before the write-ahead log recording the position of the global consistency point is saved to the persistent storage;

[0060] Use the function of the second sub-mechanism to control all common data shards to save the write-ahead log location of the global consistency point to their respective persistent storage;

[0061] Use the function of the third sub-mechanism to control all common data shards to stop playing after the write-ahead log is played to the position recorded at the global consistency point;

[0062] Use the second sub-mechanism to save the write-ahead log location of the global consistency point on the local node to persistent storage;

[0063] Play the write-ahead log content on this node to the location recorded at the global consistency point.

[0064] In this embodiment, data consistency is achieved: by using global consistency points and precise control of log playback, it is ensured that the backup data can be restored to the same consistency state as the main database at any point in time; by saving location information, the last processed location can be quickly located to reduce recovery time and complexity; by precisely controlling the playback range, the use of system resources can be optimized and the efficiency of backup and recovery can be improved; users can configure the time interval for creating global consistency points as needed and flexibly control the backup frequency; through the coordination of the three sub-mechanisms, the start and end positions of log playback can be precisely controlled to ensure data consistency and integrity; through global distributed write locks and persistent storage of location information, it is ensured that no data corruption or inconsistency occurs when processing global consistency points.

[0065] Therefore, through the above mechanism, the metadata shard node at the backup target end can not only control its own write-ahead log playback, but also coordinate and control the write-ahead log playback of all common data shard nodes, ensuring that the data of all data shards are eventually consistent during the backup process. This process ensures that the backup data can be restored to the same consistency state as the primary database at any point in time, improving the reliability and efficiency of backup and recovery.

[0066] In one embodiment, when the backup target end is restored to a common distributed database cluster, clearing invalid logs and playing to the most recent global consistency point includes:

[0067] When restoring to a common distributed database cluster on a common data sharding node at the backup target end, a cleanup mechanism for invalid write-ahead logs is performed;

[0068] When restoring to a common distributed database cluster on all data shard nodes of the backup target end, a cleanup and playback mechanism of invalid write-ahead logs is performed to play to the nearest global consistency point.

[0069] Specifically, the cleaning mechanism includes: if the transaction in which the data operation recorded in the write-ahead log is located cannot be found in the metadata shard, the write-ahead log record is discarded and no longer played;

[0070] The cleaning and playing mechanism includes: when the node is restored, all the pre-written logs after the location where the second sub-mechanism is stored in the persistent storage of this node are cleared; when the node is restored, the pre-written log records are played to the location where the second sub-mechanism is stored in the persistent storage of this node.

[0071] In this embodiment, a pre-written log cleaning mechanism for common data sharding nodes is set to ensure that only valid transaction log records are played, avoiding invalid or rolled-back transactions from affecting data consistency and preventing data inconsistency; discarding invalid log records can save storage space, reduce storage overhead, and improve storage efficiency. Thus, it is ensured that the recovered data only contains the results of valid transactions, avoiding data pollution; reducing unnecessary log processing, increasing recovery speed, and improving performance.

[0072] A pre-written log cleaning and playback mechanism is set up for all data shard nodes. By cleaning invalid log records, it can ensure that invalid log records that may cause data inconsistency will not be played during the recovery process, ensuring data consistency; simplifying the recovery process and reducing recovery time. By playing valid log records, it is ensured that all valid log records are correctly applied to the data content file during the recovery process and restored to the state at the time of backup; through the location information saved by the second sub-mechanism, the scope of log playback can be accurately controlled to ensure data consistency and improve the reliability of recovery; by accurately controlling the scope of log playback, the efficiency of recovery is improved and the recovery time is reduced.

[0073] Therefore, through the above mechanism, when the ordinary data shard nodes and all data shard nodes at the backup target end are restored to the ordinary distributed database cluster, they can effectively clean up invalid log records and accurately control the log playback range to ensure the final consistency of the data. This process not only improves the reliability and integrity of data recovery, but also optimizes the use of system resources and improves the efficiency of recovery.

[0074] The embodiments of the present application adopt a method of periodically creating global consistency points at the source end of the distributed database backup, and controlling the data synchronization progress of each data shard based on the information of these global consistency points at the backup end, to ensure that the backup data can continuously maintain the ultimate consistency of the data content in the distributed database, so that it can be quickly restored to a readable and writable distributed database cluster.

[0075] In this embodiment, a computer device is provided, such as Figure 2 As shown, it includes a memory 201, a processor 202, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above methods for continuously maintaining the final consistency of backup data is implemented.

[0076] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0077] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned methods for continuously maintaining the final consistency of backup data.

[0078] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0079] Based on the same inventive concept, an apparatus for continuously maintaining the final consistency of backup data is also provided in an embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by the apparatus for continuously maintaining the final consistency of backup data is similar to that of the method for continuously maintaining the final consistency of backup data, the implementation of the apparatus for continuously maintaining the final consistency of backup data can refer to the implementation of the method for continuously maintaining the final consistency of backup data, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the apparatus described in the following embodiments is preferably implemented in software, the implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0080] Figure 3 is a structural block diagram of a device for continuously maintaining final consistency of backup data according to an embodiment of the present invention, such as Figure 3 As shown, it includes a setting module 301, a global consistency point creation module 302, a backup module 303 and a recovery module 304. The structure is described below.

[0081] A setting module 301 is used for the distributed database to perform data backup using a write-ahead log streaming backup method, setting the data source side of the streaming backup to a backup source end, and setting the storage side of the backup data to a backup target end;

[0082] A global consistency point creation module 302, configured to create a global consistency point in the distributed database cluster of the backup source according to a preset time interval, wherein the global consistency point is information stored in a write-ahead log;

[0083] The backup module 303 is used for each data shard to continuously receive the pre-written log stream in the distributed database cluster of the backup target end, and continuously save it to the persistent storage, and ensure that the log is stopped and played at the global consistency point through a preset control mechanism;

[0084] The recovery module 304 is used to clean up invalid logs and play to the nearest global consistency point to achieve final data consistency when the backup target end is restored to a common distributed database cluster.

[0085] In one embodiment, the global consistency point creation module 302 is further configured to:

[0086] Acquire the global distributed write lock;

[0087] Write a pre-written log record of the global consistency point in all common data shard nodes of the distributed database, and send the location of this record to the node where the metadata shard is located;

[0088] Write the pre-written log record of the global consistency point to the node where the metadata shard is located, and save the location information and the shard to which the pre-written log records in all the common data shard nodes are obtained;

[0089] Release the global distributed write lock.

[0090] In one embodiment, the backup module 303 is further configured to:

[0091] On the node of the common data shard of the backup target end, the write-ahead log is continuously played as a data content file through a first preset control mechanism;

[0092] On the node of the metadata shard of the backup target end, the second preset control mechanism is used to continuously play the write-ahead log as a data content file, save the specified write-ahead log location information to persistent storage, and control the playback of the write-ahead log of the ordinary data shard of the backup target end.

[0093] In one embodiment, the backup module 303 is further configured to:

[0094] The first preset control mechanism includes a first sub-mechanism, a second sub-mechanism and a third sub-mechanism,

[0095] The first sub-mechanism is configured to stop the playback behavior before saving the write-ahead log at the specified location to the persistent storage;

[0096] The second sub-mechanism is configured to save the specified write-ahead log location information to persistent storage;

[0097] The third sub-mechanism is configured to start the playback behavior and stop the playback behavior after the write-ahead log at the specified position is played as a data content file.

[0098] In one embodiment, the backup module 303 is further configured to:

[0099] The second preset control mechanism includes a loop of the following steps, and the loop is set at the node where the metadata shard is located:

[0100] Stop the playback of write-ahead logs;

[0101] Stream and parse the write-ahead log content from the stop playback position until the next global consistency point content is found and extracted;

[0102] Parse the content of the global consistency point record and obtain the location of each record of the global consistency point in the write-ahead log of all common data shards;

[0103] Use the function of the first sub-mechanism to control all common data shards to stop playing before the write-ahead log recording the position of the global consistency point is saved to the persistent storage;

[0104] Use the function of the second sub-mechanism to control all common data shards to save the write-ahead log location of the global consistency point to their respective persistent storage;

[0105] Use the function of the third sub-mechanism to control all common data shards to stop playing after the write-ahead log is played to the position recorded at the global consistency point;

[0106] Use the second sub-mechanism to save the write-ahead log location of the global consistency point on the local node to persistent storage;

[0107] Play the write-ahead log content on this node to the location recorded at the global consistency point.

[0108] In one embodiment, the recovery module 304 is further configured to:

[0109] When restoring to a common distributed database cluster on a common data sharding node at the backup target end, a cleanup mechanism for invalid write-ahead logs is performed;

[0110] When restoring to a common distributed database cluster on all data shard nodes of the backup target end, a cleanup and playback mechanism of invalid write-ahead logs is performed to play to the nearest global consistency point.

[0111] In one embodiment, the recovery module 304 is further configured to:

[0112] The cleaning mechanism includes: if the transaction in which the data operation recorded in the write-ahead log is located cannot be found in the metadata shard, the write-ahead log record is discarded and no longer played;

[0113] The cleaning and playing mechanism includes: when the node is restored, all the pre-written logs after the location where the second sub-mechanism is stored in the persistent storage of this node are cleared; when the node is restored, the pre-written log records are played to the location where the second sub-mechanism is stored in the persistent storage of this node.

[0114] The embodiments of the present invention achieve the following technical effects: the present application adopts a method of periodically creating global consistency points at the source end of the distributed database backup, and controlling the data synchronization progress of each data shard based on the information of these global consistency points at the backup end. During recovery, by clearing invalid logs and accurately controlling log playback, it is ensured that the data is restored to a state consistent with the main database, and that the backup data can continue to maintain the ultimate consistency of the data content in the distributed database, thereby effectively improving the speed of recovery to a readable and writable distributed database cluster.

[0115] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for continuously maintaining the final consistency of backup data, characterized in that: The method comprises: Distributed databases use a write-ahead log streaming backup method to back up data. The data source side of the streaming backup is set as the backup source side, and the storage side of the backup data is set as the backup target side. In the distributed database cluster of the backup source, a global consistency point is created according to a preset time interval, and the global consistency point is information stored in a write-ahead log; the creation of the global consistency point includes: obtaining a global distributed write lock; writing a write-ahead log record of the global consistency point in all common data shard nodes of the distributed database, and sending the location of the record to the node where the metadata shard is located; writing a write-ahead log record of the global consistency point in the node where the metadata shard is located, and the record stores the location information of the write-ahead log records in all common data shard nodes obtained and the shards to which they belong; releasing the global distributed write lock; In the distributed database cluster of the backup target end, each data shard continuously receives the pre-written log stream and continuously saves it to persistent storage, and ensures that the log is stopped and played at the global consistency point through a preset control mechanism; When the backup target is restored to a common distributed database cluster, invalid logs are cleared and played to the nearest global consistency point to achieve final data consistency.

2. The method for continuously maintaining the final consistency of backup data according to claim 1, characterized in that: The method of ensuring that the log is stopped and played at the global consistency point by using a preset control mechanism includes: On the node of the common data shard of the backup target end, the write-ahead log is continuously played as a data content file through a first preset control mechanism; On the node of the metadata shard of the backup target end, the second preset control mechanism is used to continuously play the write-ahead log as a data content file, save the specified write-ahead log location information to persistent storage, and control the playback of the write-ahead log of the ordinary data shard of the backup target end.

3. The method for continuously maintaining the final consistency of backup data according to claim 2, characterized in that: The first preset control mechanism includes a first sub-mechanism, a second sub-mechanism and a third sub-mechanism, The first sub-mechanism is configured to stop the playback behavior before saving the write-ahead log at the specified location to the persistent storage; The second sub-mechanism is configured to save the specified write-ahead log location information to persistent storage; The third sub-mechanism is configured to start the playback behavior and stop the playback behavior after the write-ahead log at the specified position is played as a data content file.

4. The method for continuously maintaining the final consistency of backup data according to claim 3, characterized in that: The second preset control mechanism includes a loop of the following steps, and the loop is set at the node where the metadata shard is located: Stop the playback of write-ahead logs; Stream and parse the write-ahead log content from the stop playback position until the next global consistency point content is found and extracted; Parse the content of the global consistency point record to obtain the location of each record of the global consistency point in the write-ahead log of all common data shards; Use the function of the first sub-mechanism to control all common data shards to stop playing before the write-ahead logs recording the locations of the global consistency points are saved to persistent storage; Using the function of the second sub-mechanism to control all common data shards to save the write-ahead log location of the global consistency point to their respective persistent storages; Use the function of the third sub-mechanism to control all common data shards to stop playing after the write-ahead log is played to the position recorded at the global consistency point; Using the second sub-mechanism to save the write-ahead log location of the global consistency point on the local node to persistent storage; Play the write-ahead log content on this node to the global consistency point record location.

5. The method for continuously maintaining the final consistency of backup data according to claim 3, characterized in that: When the backup target end is restored to a common distributed database cluster, the invalid logs are cleared and played to the nearest global consistency point, including: When restoring to a common distributed database cluster on a common data sharding node at the backup target end, a cleanup mechanism for invalid write-ahead logs is performed; When restoring to a common distributed database cluster on all data shard nodes of the backup target end, a cleanup and playback mechanism of invalid write-ahead logs is performed to play to the nearest global consistency point.

6. The method for continuously maintaining the final consistency of backup data according to claim 5, characterized in that: The cleaning mechanism includes: if the transaction in which the data operation recorded in the write-ahead log is located cannot be found in the metadata shard, the write-ahead log record is discarded and no longer played; The cleaning and playing mechanism includes: when the node is restored, all the pre-written logs after the location where the second sub-mechanism is stored in the persistent storage of this node are cleared; when the node is restored, the pre-written log records are played to the location where the second sub-mechanism is stored in the persistent storage of this node.

7. A device for continuously maintaining final consistency of backup data, characterized in that: include: A setting module is used for distributed databases to perform data backup using a pre-written log streaming backup method, setting the data source side of the streaming backup as the backup source end, and setting the storage side of the backup data as the backup target end; A global consistency point creation module is used to create a global consistency point in the distributed database cluster of the backup source end according to a preset time interval, wherein the global consistency point is information stored in a write-ahead log; the creation of the global consistency point includes: obtaining a global distributed write lock; writing a write-ahead log record of the global consistency point in all ordinary data shard nodes of the distributed database, and sending the location of the record to the node where the metadata shard is located; writing a write-ahead log record of the global consistency point in the node where the metadata shard is located, wherein the record stores the location information of the write-ahead log records in all ordinary data shard nodes obtained and the shards to which they belong; releasing the global distributed write lock; A backup module, which is used for each data shard to continuously receive the pre-written log stream in the distributed database cluster of the backup target end, and continuously save it to persistent storage, and ensure that the log is stopped and played at the global consistency point through a preset control mechanism; The recovery module is used to clean up invalid logs and play to the nearest global consistency point to achieve final data consistency when the backup target end is restored to a common distributed database cluster.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for continuously maintaining the final consistency of backup data according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method for continuously maintaining the final consistency of backup data according to any one of claims 1 to 6.

Images