Data processing method and system and computing device cluster

Abstract

The invention provides a data processing method and system and a computing device cluster, and belongs to the technical field of databases. The time attribute is introduced into the write record of the preset log WAL file, so that the data can be recovered to any time point. The method comprises the steps that data of a database is fully backed up to a backup space; after full backup, the WAL file is subjected to incremental backup to a backup space, the WAL file is used for storing a write record, and the write record comprises data and a timestamp of the data; when the data of the database needs to be recovered to the target time point, obtaining full-amount backup data and an incremental backup target WAL file from the backup space; and on the basis of recovering the total backup data, by comparing the target time point with the timestamp of the data in the write record of the target WAL file, playing back the write record of which the timestamp of the data in the target WAL file is located before the target time point so as to recover the data of the database to the target time point.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 26, 2023, with application number 202311083412.1 and application name “A data recovery method, device and computing device cluster”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of database technology, and in particular to a data processing method, system and computing device cluster. Background Art

[0003] When using a database, it is usually necessary to back up the data so that data can be restored in the event of a series of abnormal scenarios such as data corruption, data loss, and data deletion caused by user error. Currently, when backing up data, incremental data is usually copied and stored in the backup space at several consecutive backup time points at a certain time interval. However, when restoring data, users can only select a certain backup time point to restore the data, and cannot restore the data to any time between two backup time points. Therefore, how to restore the database to any time point is an important technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a data processing method, system and computing device cluster, which introduces a time attribute in the write record of a preset log WAL file, so that when restoring data, the time attribute in each write record can be used as a basis, so that the data can be restored to any time point.

[0005] To achieve the above objectives, the present application provides the following technical solutions:

[0006] In a first aspect, a data processing method is provided, which can be applied to a database system, wherein the database system uses a write-ahead log (WAL) to record operations on the database, and the method comprises: backing up the data of the database in full to a backup space; after the full backup, incrementally backing up the WAL file to the backup space, wherein the WAL file is used to store the write record, and the write record includes data and a timestamp of the data; when it is necessary to restore the data of the database to a target time point, obtaining the data of the full backup and the target WAL file of the incremental backup from the backup space, wherein the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, and the target incremental backup is the most recent incremental backup after the target time point; on the basis of restoring the data of the full backup, by comparing the timestamps of the data in the write record of the target WAL file with the timestamps of the data at the target time point, replaying the write record in which the timestamp of the data in the target WAL file is before the target time point, so as to restore the data of the database to the target time point.

[0007] As can be seen from the above, the embodiment of the present application introduces a time attribute in the write record of the WAL file, that is, each write record corresponds to its own timestamp. When backing up the database log file, the database system first performs a full backup, and then incrementally backs up the WAL file with the time attribute. In this way, each write record in the WAL file sent by the database system to the backup space is timestamped. When it is necessary to restore the data of the database to the target time point, the database system can first roughly obtain the data of the full backup and the target WAL file of the incremental backup from the backup space according to the backup time. Among them, the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, and the target incremental backup is the most recent incremental backup after the target time point. Further, on the basis of restoring the data of the full backup, by comparing the timestamps of the data in the write records of the target time point and the target WAL file, the write records whose timestamps of the data in the target WAL file are located before the target time point are played back to restore the data of the database to the target time point, thereby improving the time accuracy of data recovery.

[0008] In a possible implementation, the method further includes: receiving a write request from a client; in response to the write request, writing data corresponding to the write request and a timestamp of the data corresponding to the write request into a WAL file to obtain a write record corresponding to the write request.

[0009] From the above, we can see that after receiving a write request, the database system not only writes the data corresponding to the write request into the WAL file, but also writes the timestamp of the data corresponding to the write request into the WAL file, so that the write record corresponding to the write request stored in the WAL file has a time attribute.

[0010] In a possible implementation, the method further includes: storing the full WAL file to a directory to be backed up; and incrementally backing up the WAL file to the backup space, including: incrementally backing up the WAL file in the directory to be backed up to the backup space.

[0011] From the above, we can see that after storing the full WAL file in the directory to be backed up, the WAL files in the directory to be backed up are backed up to ensure that the newly generated WAL files can be backed up in time, providing strong support for data recovery.

[0012] In a possible implementation, the method further includes: deleting the WAL files that have been incrementally backed up to the backup space.

[0013] From the above, we can see that timely deleting the backed-up WAL files can reduce the space occupied by a large number of WAL files in the database.

[0014] In a second aspect, a data processing method is provided, which can be applied to a database system. The database system uses a write-ahead log (WAL) to record operations on the database. The method includes: backing up the database data in full to a backup space; after the full backup, incrementally backing up the WAL file to the backup space, wherein the WAL file is used to store write records, and the write records include data and timestamps of the data; after the full backup, incrementally backing up the database data to the backup space, wherein at least one incremental backup of the WAL file is performed between every two incremental backups of the data; when the database data needs to be restored to a target time point, obtaining the full backup data, the incremental backup target data, and the incremental backup target WAL from the backup space. file; wherein, the target data includes the data of the first incremental backup and the data of the incremental backup before the first incremental backup, the first incremental backup is the incremental backup of the data most recently before the target time point, the target WAL file includes the WAL file of the second incremental backup, the WAL file before the second incremental backup and after the first incremental backup, and the second incremental backup is the incremental backup of the WAL file most recently after the target time point; on the basis of restoring the data of the full backup and the target data of the incremental backup, by comparing the timestamps of the data in the write records of the target time point and the target WAL file, the write records whose timestamps of the data in the target WAL file are located before the target time point are played back to restore the data of the database to the target time point.

[0015] From the above, we can see that since the database system adopts the method of interspersed incremental backup of database data during the incremental backup of WAL files, the number of target WAL files that the database system needs to obtain from the backup space is significantly reduced, thereby improving the playback speed and facilitating data recovery.

[0016] In a third aspect, a data processing method is provided, which can be applied to a database system, wherein the database system uses a write-ahead log (WAL) to record operations on the database, and the method comprises: backing up the database data in full to a backup space; after the full backup, incrementally backing up the WAL file and metadata information to the backup space, wherein the WAL file is used to store write records, the write record includes data and a timestamp of the data, and the metadata information includes metadata and a timestamp of the metadata; when the database data and metadata need to be restored to a target time point, obtaining the full backup data, the incremental backup target WAL file, and the incremental backup target metadata information from the backup space, wherein the target WAL file includes the target incremental backup WAL file and the target incremental backup The WAL file of the previous incremental backup is backed up. The target incremental backup is the latest incremental backup of the WAL file after the target time point. The target metadata information includes the metadata information of the target incremental backup and the metadata information of the incremental backup before the target incremental backup. On the basis of restoring the data of the full backup, by comparing the timestamps of the data in the write records of the target WAL file with the target time point, the write records with the timestamps of the data in the target WAL file before the target time point are replayed to restore the data of the database to the target time point. By comparing the timestamps of the metadata in the target metadata information with the timestamps of the metadata in the target metadata information, the metadata with the timestamps before the target time point is restored to restore the metadata of the database to the target time point.

[0017] As can be seen from the above, the database system adds timestamps to the above metadata information when backing up data, and backs up metadata information with timestamps. When the database data and metadata need to be restored to the target time point, the metadata before the target time point can be restored, and these metadata can make the restored data structure consistent with the original data structure in the database.

[0018] In a fourth aspect, a data processing system is provided, the system comprising: a backup device, the backup device being used to: fully back up the data of a database to a backup space; after the full backup, incrementally back up WAL files to the backup space, wherein the WAL files are used to store write records, and the write records include data and timestamps of the data; a recovery device, the recovery device being used to: when it is necessary to restore the data of the database to a target time point, obtain the data of the full backup and the target WAL files of the incremental backup from the backup space, the target WAL files including the WAL files of the target incremental backup and the WAL files of the incremental backup before the target incremental backup, the target incremental backup being the most recent incremental backup after the target time point; on the basis of restoring the data of the full backup, by comparing the timestamps of the data in the write records of the target WAL file with the timestamps of the data at the target time point, replay the write records whose timestamps of the data in the target WAL file are before the target time point, so as to restore the data of the database to the target time point.

[0019] In one possible implementation, the system also includes: a receiving device, the receiving device is used to receive a write request from a client; a responding device, the responding device is used to respond to the write request, write the data corresponding to the write request and the timestamp of the data corresponding to the write request into the WAL file to obtain a write record corresponding to the write request.

[0020] In a possible implementation, the backup device is also used to store the full WAL file to the directory to be backed up; the backup device is used to incrementally back up the WAL file to the backup space, including: the backup device is used to incrementally back up the WAL file in the directory to be backed up to the backup space.

[0021] In a possible implementation, the backup device is further used to: delete the WAL files that have been incrementally backed up to the storage space.

[0022] In a fifth aspect, a data processing system is provided, the system comprising: a backup device, the backup device being used to: fully back up the data of a database to a backup space; after the full backup, incrementally back up the WAL file to the backup space, wherein the WAL file is used to store write records, and the write records include data and timestamps of the data; after the full backup, incrementally back up the data of the database to the backup space, wherein at least one incremental backup of the WAL file is performed between every two incremental backups of the data; a recovery device, the recovery device being used to: when it is necessary to restore the data of the database to a target time point, obtain the data of the full backup, the target data of the incremental backup, and the target WAL file of the incremental backup from the backup space; wherein, The target data includes the data of the first incremental backup and the data of the incremental backup before the first incremental backup, the first incremental backup is the latest incremental backup of the data before the target time point, the target WAL file includes the WAL file of the second incremental backup, the WAL file before the second incremental backup and after the first incremental backup, the second incremental backup is the latest incremental backup of the WAL file after the target time point; on the basis of restoring the data of the full backup and the target data of the incremental backup, by comparing the timestamps of the data in the write records of the target time point and the target WAL file, the write records with the timestamps of the data in the target WAL file before the target time point are played back to restore the data of the database to the target time point.

[0023] In a sixth aspect, a data processing system is provided, the system comprising: a backup device, the backup device being used to: fully back up the data of a database to a backup space; after the full backup, incrementally back up WAL files and metadata information to the backup space, wherein the WAL files are used to store write records, the write records include data and a timestamp of the data, and the metadata information includes metadata and a timestamp of the metadata; a recovery device, the recovery device being used to: when it is necessary to restore the data and metadata of the database to a target time point, obtain the data of the full backup, the target WAL file of the incremental backup, and the target metadata information of the incremental backup from the backup space, wherein the target WAL file includes the WAL file of the target incremental backup and the incremental information before the target incremental backup The target incremental backup is the latest incremental backup of the WAL file after the target time point, and the target metadata information includes the metadata information of the target incremental backup and the metadata information of the incremental backup before the target incremental backup; on the basis of restoring the data of the full backup, by comparing the target time point with the timestamps of the data in the write records of the target WAL file, the write records with the timestamps of the data in the target WAL file before the target time point are played back to restore the data of the database to the target time point; by comparing the timestamps of the target time point with the timestamps of the metadata in the target metadata information, the metadata with the timestamps before the target time point in the target metadata information is restored to restore the metadata of the database to the target time point.

[0024] In the seventh aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the data processing method as described in the first to third aspects above.

[0025] In an eighth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the data processing method of the first to third aspects described above.

[0026] In a ninth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the data processing method of the first to third aspects described above.

[0027] It should be noted that various possible implementation methods of any one of the above aspects can be combined under the premise that there is no contradiction between the solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a database system provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the structure of a database provided in an embodiment of the present application;

[0030] Figure 3 One of the flow charts of the data processing method provided in the embodiment of the present application;

[0031] Figure 4 One of the backup method schematic diagrams provided in the embodiment of the present application;

[0032] Figure 5 A Record encoding schematic diagram provided for an embodiment of the present application;

[0033] Figure 6 A schematic diagram of a Record structure provided in an embodiment of the present application;

[0034] Figure 7 Another Record encoding schematic diagram provided for an embodiment of the present application;

[0035] Figure 8 A schematic diagram of RocksDB managing WAL files provided in an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of a data recovery process provided in an embodiment of the present application;

[0037] Fig.10 A schematic diagram of data playback provided in an embodiment of the present application;

[0038] Fig.11 One of the flow charts of the data processing method provided in the embodiment of the present application;

[0039] Fig.12 One of the backup method schematic diagrams provided in the embodiment of the present application;

[0040] Fig.13 One of the flow charts of the data processing method provided in the embodiment of the present application;

[0041] Fig.14 A schematic diagram of data that needs to be backed up provided in an embodiment of the present application;

[0042] Fig.15 A schematic diagram of metadata information recovery provided in an embodiment of the present application;

[0043] Fig.16 Another schematic diagram of metadata information recovery provided in an embodiment of the present application;

[0044] Fig.17 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0045] Fig.18 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;

[0046] Fig.19 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0047] Fig. 20 A schematic diagram of the structure of a computer cluster provided in an embodiment of the present application;

[0048] Fig.21 A schematic diagram of the structure of another computer cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solution provided by the present application will be described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0050] In the description of the embodiments of the present application, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0051] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0052] In order to make the technical solution provided by the present application clearer, before describing the technical solution provided by the present application in detail, some relevant terms and related technologies involved in the embodiments of the present application are first introduced.

[0053] Key-Value (KV) storage system: a storage system that stores data in key-value pairs, similar to the map in Java. The entire database can be understood as a large map, and each key corresponds to a unique value. The system can take values ​​based on the primary key (also known as the keyword or primary code), and can query data through the primary key. It has fast query speed, large data storage volume, and supports high concurrency. Among them, the primary key can be composed of one field or multiple fields, which are called single-field primary keys or multi-field primary keys respectively. The primary key can uniquely identify a record (an entry) in a file. In a database that uses the KV storage system, when a new record is added, the database can automatically check the primary key value of the new record.

[0054] RocksDB: is a KV storage system based on log structured merge tree (LSMTree). The architecture of RocksDB includes: memory table (MemTable), which is used to store recently written data; persistent storage table (SSTable), which is used to store data that has been written to disk; log file (Log File), also known as write ahead log (WAL), which is used to record all write operations; compaction (Compaction), which is used to merge multiple SSTables into a larger SSTable. When writing in RocksDB, the data will first be written to WAL and MemTable in memory. When MemTable is full, it will be converted into an immutable memory table (ImmutableMemTable). The data in Immutable MemTable will eventually be written to the data file SSTable on persistent storage (such as disk). The data files on the disk are divided into many layers. The higher the layer, the fewer files, and the lower the layer, the more files. When a layer is full, the background compaction thread will be triggered to merge the data files of the upper layer into the next layer. After the data is merged into the next layer, the SSTable files of this layer can be deleted.

[0055] Record: Each write statement is converted into a Record and written into the WAL file. During the recovery process, the Records in the WAL are read sequentially for recovery.

[0056] Database backup: Database backup refers to backing up the data and structure in the database to another location so that the data can be restored when needed. Database backup is usually used to prevent data loss and to recover when the database fails.

[0057] Full backup: also known as full backup, refers to a complete copy of all the data in the database at a certain point in time. In actual applications, a tape can be used to fully back up the data of the entire database.

[0058] Incremental Backup: Incremental backup means that after a full backup or the last incremental backup, each subsequent backup needs to back up files that have been added or modified compared to the previous one. Specifically, in this application, the incremental backup WAL file refers to the WAL file that needs to be backed up for each backup compared to the previous one; in this application, the incremental backup data refers to the data that needs to be backed up for each backup compared to the previous one.

[0059] In some abnormal scenarios (such as data corruption, data loss, and data deletion caused by user error), users often hope that the database can be quickly restored to a certain time before the abnormality occurs to minimize losses, and the accuracy requirement is at the second level. Therefore, the ability to restore the database to any point in time is particularly important.

[0060] However, the RocksDB storage system has no time attribute. When backing up and restoring a database based on the RocksDB storage system, a full backup based on the data is usually performed first, all data is completely copied, and uploaded to the cloud space. After that, an incremental backup based on the data is performed at regular intervals. The incremental backup of the data will upload the incremental and differential files, and finally a full backup time point and continuous incremental backup time points that are maintained at a certain interval will be obtained. When users need to restore data, they can only select the above time points to restore data.

[0061] Therefore, the related technology may have the following problems when restoring data: On the one hand, since the RocksDB storage system has no time attribute, users can only select some fixed backup time points for recovery when restoring data, and cannot restore to any time point. On the other hand, since the RocksDB storage system will periodically compress (Compaction) the data, it may cause the files in the storage system to change, but the actual data has not changed. The differential backup will think that the data has changed before and after the compaction operation, and upload the differential file, which will cause the data upload amount to be greater than the actual change amount.

[0062] In view of the above problems, the embodiments of the present application provide a data processing method, device, equipment and storage medium. In the data backup stage, by introducing a time attribute in the write record of a preset log WAL file, the data can be restored to any time point according to the time attribute in each write record during data recovery. In addition, the present application only backs up the WAL files and some meta information in the database, which can avoid uploading a large amount of redundant data during backup.

[0063] The data processing method provided by the embodiment of the present application is described in detail below in conjunction with the accompanying drawings. It should be noted that the embodiments of the present application can refer to each other, for example, the same or similar steps, method embodiments, system embodiments and device embodiments can refer to each other without limitation.

[0064] First, an example application scenario of the data processing method provided in the embodiment of the present application is introduced. The data processing method provided in the embodiment of the present application can be applied to Figure 1 The database system shown, Figure 1 A schematic diagram of the database system architecture is given. Figure 1 As shown, the database system at least includes database instances (such as database instance 1, database instance 2), a storage engine, a storage layer, and a backup space. Among them, database instance 1 is the instance corresponding to the original database, recorded as the original instance. Database instance 2 is the instance corresponding to the new database, recorded as the new instance. The original database can be any database using a KV storage system, such as but not limited to RocksDB, LevelDB, etc. The database system can use WAL to record operations on the database.

[0065] In some embodiments, an external client (such as client 1, client 2, client n) may send a request to database instance 1 to request access to the original database (for example, to write or delete data). Further, in response to the user's write request, the storage engine of the original database may drive the data to be stored in the storage layer to implement the user's write request. The storage layer may be any storage medium, such as a disk, an optical disk, etc.

[0066] Specifically, Figure 2As shown, the database (including the original database and the new database) can be used to accept client access, manage SST files, WAL, manifest files and current files, so as to provide users with functions such as reading (query) and writing (including adding (PUT) / deleting (DELETE) and other operations). Among them, WAL is a log used to record the content of operations in the process of processing data insertion and deletion. Before each business data write operation is executed, it will be recorded in WAL and written to MemTable at the same time. The Manifest file records the metadata of each SST file, including information such as file name, maximum primary key and minimum primary key. The Current file is used to record the file name of the current Manifest file of the database.

[0067] The database can also manage SST files in layers, that is, all SST files form a hierarchical structure. Each layer can maintain a specified number or size of files. For example, the 0th layer is level 0, which can include 1M SST files, the 1st layer is level 1, which can include 10M SST files, the 2nd layer is level 2, which can include 100M SST files, the 3rd layer is level 3, which can include 1G SST files, and so on. Among them, level 0-n represents the different levels of SST file storage, and the level sequence number from small to large represents the data in the database from new to old (that is, the time when the database receives data from early to late). The SST files in each layer can be sorted according to the size of the primary key. Each SST file has the smallest primary key and the largest primary key, and the primary keys of each SST file do not overlap. Therefore, to find a primary key in a layer, only one SST file needs to be found. The level information to which each SST file belongs can be recorded in the Manifest file.

[0068] In some embodiments, the database system can also back up the data of the original database in the storage layer, for example, to a specific backup space. The specific data backup process can refer to the introduction in the following method embodiment, which will not be repeated here. Among them, the backup space can be any space, such as cloud space, distributed shared storage platform, etc.

[0069] In some embodiments, the database system can also restore the data of the original database in the backup space to obtain a new database instance. For example, in response to a user's data recovery request, the database system can restore the backup data in the cloud space to a specific time point to obtain a database instance 2. The specific data recovery process can refer to the introduction in the following method embodiment, which will not be repeated here.

[0070] It should be noted that the embodiments of the present application do not limit specific application scenarios. The system architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It is known to those skilled in the art that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0071] The following is an introduction to the data processing method provided in the embodiments of the present application.

[0072] Figure 3 A flow chart of a data processing method provided in an embodiment of the present application. Figure 3 , the method may include the following steps:

[0073] S201. Back up all the data in the database to the backup space.

[0074] As a possible implementation method, the database system can perform a full backup of the data in the database at the initial backup time, and upload all current data in the database to the backup space.

[0075] It should be noted that the initial backup time can be set in advance by the operation and maintenance personnel for the database system. For example, the initial backup time can be the time when the database system is started for a full week. The backup space can be any storage space configured by the operation and maintenance personnel for the database system, such as cloud space, distributed shared storage platform, etc.

[0076] Exemplarily, when the initial backup time is reached, the database system may use a database backup tool to automatically back up all data in the database to the cloud space.

[0077] As another possible implementation, in response to the backup instruction, the database system performs a full backup of the data in the database, and uploads all current data in the database to the backup space.

[0078] Exemplarily, the operation and maintenance personnel may issue a backup instruction to the database system, and the database system responds to the backup instruction to perform a full backup of the data in the database, and copies all current data in the database to the cloud space.

[0079] S202: After the full backup, incrementally back up the WAL file to the backup space.

[0080] Among them, the WAL file is used to store write records, which include data and the timestamp of the data.

[0081] As a possible implementation method, after a full backup, the database system can periodically perform incremental backup of WAL files to the backup space.

[0082] As another possible implementation, after the full backup, the database system can non-periodically incrementally back up the WAL files to the backup space.

[0083] For example, after the full backup, the database system can perform an incremental backup of the WAL file at a certain interval and upload the incremental WAL file to the cloud space. The time interval can be a fixed time interval (i.e., periodic) or a non-fixed time interval (i.e., non-periodic).

[0084] Specifically, after completing a full backup, the storage device can perform a log-based incremental backup (i.e., incremental backup WAL file) at regular intervals. For example, at the first time interval after the full backup, the database system can first determine the newly added WAL files in the database at that time compared to the WAL files in the database at the time of the full backup. Further, the database system transfers the newly added WAL files to the backup space. At the second time interval (i.e., a period of time after the first time interval), the database system determines the newly added WAL files in the database between the second time interval point and the first time interval point, and transfers the newly added WAL files to the backup space.

[0085] like Figure 4 As shown in the figure, the database system can adopt the backup method of full backup + log incremental backup. The database system performs a full backup of the data in the database at the initial backup time ( Figure 4 Db1 is used to represent the full amount of data, and Db1 includes all the data in the database at the time of data backup), and the current data is fully uploaded to the cloud space. After that, the database system performs a log-based incremental backup at regular intervals (i.e. Figure 4 Log1, ..., Log11, ..., where each Log represents a WAL file added to the database within a time interval).

[0086] It should be noted that the write record of the WAL file includes not only the written data, but also the timestamp of the data. The timestamp of the data adds a time attribute to the data, and optionally, the timestamp can be used to indicate the writing time of the data.

[0087] In actual applications, when the client stores data in the database, the database can convert each write statement into a record and write it into the WAL file.

[0088] For example, the database system receives a write request from a client. In response to the write request, the database system writes the data corresponding to the write request and the timestamp of the data corresponding to the write request into a WAL file to obtain a write record corresponding to the write request.

[0089] In some embodiments, the data corresponding to the write request includes a key key and a value value, and the database system includes a database instance and a storage engine (such as Figure 1 ). In order to obtain a write record with a timestamp, after the database system receives the write request from the client, the database instance can add a timestamp to the value value inside the database system to obtain the value value with a timestamp. The database instance sends the key key and the value value with a timestamp to the storage engine. Correspondingly, after receiving the key key and the value value with a timestamp sent by the database instance, the storage engine writes the key key and the value value with a timestamp to the WAL file to obtain the write record (i.e., Record) corresponding to the write request.

[0090] It should be noted that Record is the basic unit of data storage in WAL files. Data is recorded in WAL files one by one. Figure 5 As shown, RocksDB is a KV storage system. Records in RocksDB contain the key-value (KV) information of the data. Optionally, the database instance can add a timestamp to the value according to the data corresponding to the write request (including the key and value) and the writing time of the data, and then encode the Record to obtain a Record with a timestamp.

[0091] For example, the database instance can add a 4-byte long type timestamp in front of the Value to store the writing time of the data. When decoding the Record, the decoding side can first read the 4-byte long type timestamp, followed by the specific Value of the data.

[0092] like Figure 6As shown in the figure, a Record at least includes four parts: Cyclic Redundancy Check (CRC) field, Size field, Type field, and Payload field. Among them, CRC contains some information used to verify data, Size is used to reflect the data byte size of this Record, Type is used to reflect the type of Record, and Payload contains Key and Value. A piece of data can be composed of one or more Records, which depends on the size of the Value corresponding to the data. If the Value is relatively small, it may be stored in one Record (the Type of the Record can be full in this case). If the Value is relatively large, it may be split into multiple Records for storage (the Type of the first Record can be first, the Type of the middle Record can be middle (there can be multiple), and the Type of the last Record can be last). When reading, these Records are pieced together to form a KV of complete data.

[0093] It can be understood that in the above method, the database directly changes the Value when storing the KV, adds the writing time of the data to the Value, and obtains the Value with a timestamp. This method does not need to change the structure of the Record and is easy to encode and decode the Record.

[0094] In some other embodiments, in order to obtain a write record with a timestamp, after the database system receives a write request from a client, the database instance sends the key key and the value value to the storage engine within the database system. Correspondingly, after receiving the key key and the value value sent by the database instance, the storage engine determines the timestamp of the data corresponding to the write request, and writes the key key, the value value, and the timestamp of the data corresponding to the write request into the WAL file to obtain a write record corresponding to the write request.

[0095] Exemplarily, when the storage engine writes the key key, value value, and the timestamp of the data corresponding to the write request into the WAL file, a new field (recorded as the first field) can be added to the Record to record the writing time of the data.

[0096] like Figure 7 As shown, the embodiment of the present application can be Figure 6 Based on the Record structure shown in FIG. 1 , a first field is added before the Payload field to store a timestamp (i.e. Figure 7 The Timestamp part in the ), and then get the Record with the timestamp.

[0097] It is understandable that the above method changes the structure of the Record and adds a timestamp part without changing the original Value of the written data. The time attribute can be introduced into the Record without the user's perception, which has little impact on the user's use of the database.

[0098] Optionally, if the Value is relatively small and only one Record is used to store it, the storage engine only needs to add the first field to the full type Record according to the above method, and add the write time to the first field. If the Value is relatively large and needs to be stored in multiple Records, the storage engine can add the first field to the first type Record among the multiple Records according to the above method, and add the write time to the first field.

[0099] In some embodiments, in order to back up WAL files, whenever a WAL file is full of Records, the database system determines the WAL file as a WAL file to be backed up, and backs up the WAL file to a corresponding backup space, such as a cloud space, by incrementally backing up the WAL file.

[0100] In some other embodiments, in order to facilitate uploading of full WAL files to the backup space, the database system pre-creates a log directory to be backed up (referred to as the directory to be backed up), and whenever a WAL file is full of Records, the database system archives the WAL file to the log directory to be backed up. Furthermore, the database system incrementally backs up the WAL files in the directory to be backed up to the backup space.

[0101] In some related technologies, RocksDB has a log deletion function, that is, whenever a WAL file is full of Records, RocksDB usually deletes the WAL file. In the embodiment of the present application, the database system can set RocksDB to prohibit the log deletion function, so that RocksDB does not temporarily delete the WAL file that is full of Records. Optionally, when the WAL file is backed up, RocksDB can be allowed to delete the WAL file.

[0102] like Figure 8 The figure shows the process of managing WAL files by RocksDB in the embodiment of the present application. The database system sets RocksDB to prohibit deleting logs through the database interface, and instructs RocksDB to delete the WAL file filled with Records ( Figure 8During the backup process, the database system controls the upload of WAL files through the backup management function module. After the WAL files are successfully uploaded, RocksDB is instructed to delete the uploaded WAL files.

[0103] S203: When the database data needs to be restored to the target time point, the full backup data and the target WAL file of the incremental backup are obtained from the backup space.

[0104] The target WAL files include the WAL files of the target incremental backup and the WAL files of the incremental backup before the target incremental backup. The target incremental backup is the latest incremental backup after the target time point.

[0105] As a possible implementation method, when the database data needs to be restored to the target time point, the user can initiate a data recovery instruction to the database system. The data recovery instruction includes the target time point. Accordingly, in response to the data recovery instruction initiated by the user, the database system can obtain the full backup data and the target WAL file of the incremental backup from the backup space to perform data recovery.

[0106] In some embodiments, the database system can determine the most recent incremental backup after the target time point to be restored (recorded as the target incremental backup), and obtain the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup.

[0107] For example, the database system performs a full backup at time 1, performs an incremental WAL file backup at time 2 after time 1, performs an incremental WAL file backup at time 3 after time 2, and performs an incremental WAL file backup at time 4 after time 3. If the target time point is time 2.5, the database system obtains the data of the full backup at time 1, the WAL file of the incremental backup at time 2, and the WAL file of the incremental backup at time 3 from the backup space.

[0108] It is understandable that when the database data needs to be restored to a target time point, the database system must not only obtain the WAL files of the incremental backup before the target time point, but also obtain the WAL files of the most recent target incremental backup after the target time point. This ensures that the WAL files generated between the target incremental backup and the previous incremental backup are not missed, making subsequent data recovery more accurate.

[0109] In some embodiments, the database system can also display a data recovery time selection interface for the user, which can include multiple time points, and the user can select the time point to be restored in the interface. Further, the data recovery device can obtain the full backup data and the target WAL file of the incremental backup from the backup space according to the time point selected by the user.

[0110] S204. On the basis of restoring the full backup data, by comparing the target time point with the timestamps of the data in the write records of the target WAL file, replaying the write records in the target WAL file whose timestamps are before the target time point, so as to restore the database data to the target time point.

[0111] As a possible implementation method, after obtaining the full backup data and the target WAL file of the incremental backup from the backup space, the database system can restore the full backup data on the new database instance. Further, based on the restoration of the full backup data, the database system compares the timestamps of the data in the write records of the target WAL file with the timestamps of the data at the target time point, and replays the write records of the data in the target WAL file whose timestamps are before the target time point, so as to restore the database data to the target time point.

[0112] In actual applications, the database system can decode the write records in the target WAL file to obtain the timestamp corresponding to each write record. Further, the database system compares the timestamp corresponding to each write record with the target time point and plays back the write records in the target WAL file whose timestamps are before the target time point.

[0113] In some embodiments, the database system is configured with a log filter. After acquiring the target WAL file of the incremental backup from the backup space, the database system can filter the write records in the target WAL file through the log filter to play back the write records whose timestamps of the data in the target WAL file are before the target time point.

[0114] For example, the operation and maintenance personnel can set up WalFilter in the database system to customize the WalFilter module, which can inherit from the native WalFilter of RocksDB. When recovering data, the database system can use the WalFilter module to filter and replay WAL files according to the specified recovery time.

[0115] like Fig. 9As shown in the figure, after setting WalFilter in the database system, the user can specify the time to be restored in the database system (recorded as pitr time). For the target WAL file obtained from the backup space, the WalFilter filtering function is started, and the database system traverses the Record in the target WAL file and starts filtering and replaying the WAL file. Specifically, the database system parses the timestamp corresponding to each Record (recorded as Record time). When the Record time of a Record is greater than pitr time, the playback ends, and the write records whose timestamps of the data in the target WAL file are located before the target time point are obtained, and the database data is restored to the target time point according to the contents of these write records.

[0116] like Fig.10 As shown, it is assumed that Record1-Record5 are records before pitr time, and Record6-Record8 are records after pitr time. When the data recovery device plays back the log file through WalFilter, it will read the records in these log files in sequence (i.e., read record) and parse the write time corresponding to each record. When playing back to Record6, since the write time corresponding to Record6 is greater than pitr time, the data recovery device stops playing and recovers the write records of Record1-Record5.

[0117] The embodiment of the present application introduces a time attribute in the write record of the WAL file, that is, each write record corresponds to its own timestamp. When backing up the database log file, the database system first performs a full backup, and then incrementally backs up the WAL file with the time attribute. In this way, each write record in the WAL file sent by the database system to the backup space is timestamped. When it is necessary to restore the data of the database to the target time point, the database system can first roughly obtain the data of the full backup and the target WAL file of the incremental backup from the backup space according to the backup time. Among them, the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, and the target incremental backup is the most recent incremental backup after the target time point. Further, on the basis of restoring the data of the full backup, by comparing the timestamps of the data in the write record of the target time point and the target WAL file, the write record with the timestamp of the data in the target WAL file before the target time point is played back to restore the data of the database to the target time point, thereby improving the time accuracy of data recovery.

[0118] In one design, in order to avoid too many WAL files for incremental backup, Fig.11As shown, the data processing method provided in the embodiment of the present application may also include the following steps:

[0119] S301. Back up all the data in the database to the backup space.

[0120] As a possible implementation method, the database system can perform a full backup of the data in the database at the initial backup time, and upload all current data in the database to the backup space.

[0121] It should be noted that the specific implementation method of this step can refer to the above S201 and will not be repeated here.

[0122] S302: After the full backup, incrementally back up the WAL file to the backup space.

[0123] Among them, the WAL file is used to store write records, which include data and the timestamp of the data.

[0124] It should be noted that the specific implementation of this step can refer to the above S201, which will not be repeated here. The difference is that after the full backup, the database system intersperses the incremental backup of the database data while incrementally backing up the WAL file, which can be specifically referred to in the following S303.

[0125] S303: After the full backup, incrementally back up the database data to the backup space.

[0126] Among them, at least one incremental backup of the WAL file is performed between every two incremental backups of data.

[0127] As a possible implementation method, after a full backup, the database system incrementally backs up the database data at regular intervals while incrementally backing up the WAL files.

[0128] As another possible implementation, after the full backup, the database data is incrementally backed up once after the incremental backup of the WAL file is performed a preset number of times.

[0129] As another possible implementation, after the full backup, the database system incrementally backs up the WAL files to the backup space. Whenever the number of incrementally backed up WAL files reaches a preset number, the database data is incrementally backed up once.

[0130] like Fig.12As shown, the database system first performs a full data backup (Db1), and further, the database system can perform log incremental backup (Log) based on WAL files at different backup times (such as the first backup time, the second backup time, etc.). During the log incremental backup (such as the first time), the database system can intersperse the incremental backup (Diff) of the database data to send the incremental backup data to the backup space.

[0131] It is understandable that when the full backup cycle is too long, it is easy to cause a lot of WALs in the incremental log backup, and too many WAL files will cause problems such as slow log playback, log playback jamming, and recovery failure during recovery. Therefore, in the case of a long full backup cycle, the embodiment of the present application introduces incremental backup of database data to balance the problem of too many WAL logs that need to be played back.

[0132] S304: When the database data needs to be restored to the target time point, the full backup data, the target data of the incremental backup, and the target WAL file of the incremental backup are obtained from the backup space.

[0133] Among them, the target data includes the data of the first incremental backup and the data of the incremental backup before the first incremental backup. The first incremental backup is the latest incremental backup of the data before the target time point. The target WAL file includes the WAL file of the second incremental backup, the WAL file before the second incremental backup and after the first incremental backup. The second incremental backup is the latest incremental backup of the WAL file after the target time point.

[0134] In some embodiments, the database system can determine the database data of the latest incremental backup before the target time point to be restored, and obtain the incremental backup data and the data of the incremental backup before the incremental backup. Further, the database system can also determine the WAL file of the latest incremental backup after the target time point to be restored, and obtain the WAL file of this incremental backup and the WAL file backed up before the incremental backup WAL file and after the above incremental backup data.

[0135] Exemplarily, the database system performs a full backup at time 1, performs an incremental WAL file backup at time 2 after time 1, performs an incremental database data backup at time 3 after time 2, performs an incremental WAL file backup at time 4 after time 3, performs an incremental database data backup at time 5 after time 4, and performs an incremental WAL file backup at time 6 after time 5. If the target time point is time 5.5, the database system obtains the data of the full backup at time 1, the database data of the incremental backup at time 5, the database data of the incremental backup at time 3, and the WAL file of the incremental backup at time 6 from the backup space.

[0136] In contrast, if only incremental backup of WAL files is performed after full backup, the database system performs a full backup at time 1, performs an incremental WAL file backup at time 2 after time 1, performs an incremental WAL file backup at time 3 after time 2, performs an incremental WAL file backup at time 4 after time 3, performs an incremental WAL file backup at time 5 after time 4, and performs an incremental WAL file backup at time 6 after time 5. If the target time point is time 5.5, the database system obtains the data of the full backup at time 1, the WAL files of the incremental backup at time 2, the WAL files of the incremental backup at time 3, the WAL files of the incremental backup at time 4, the WAL files of the incremental backup at time 5, and the WAL files of the incremental backup at time 6 from the backup space.

[0137] It can be seen that when the database system performs incremental backup of database data during the incremental backup of WAL files, compared with the method of only incrementally backing up WAL files after the full backup, when the database data needs to be restored to the target time point, the number of target WAL files that the database system needs to obtain from the backup space is significantly reduced.

[0138] S305. On the basis of restoring the target data of the full backup and the incremental backup, by comparing the timestamps of the data in the write records of the target WAL file with the timestamps of the data at the target time point, replay the write records in the target WAL file whose timestamps are before the target time point, so as to restore the database data to the target time point.

[0139] As a possible implementation, after obtaining the full backup data, the target data of the incremental backup, and the target WAL file of the incremental backup from the backup space, the database system can restore the full backup data and the target data of the incremental backup on the new database instance. Further, based on the restored database data, the database system compares the timestamps of the data in the write records of the target WAL file with the timestamps of the data at the target time point, and replays the write records of the data in the target WAL file whose timestamps are before the target time point, so as to restore the database data to the target time point.

[0140] It is understandable that since the database system uses an interleaved incremental backup method for database data during the incremental backup of WAL files, the number of target WAL files that the database system needs to obtain from the backup space is significantly reduced, thereby increasing the playback speed and facilitating data recovery.

[0141] Fig.13 A flowchart of another data processing method provided in an embodiment of the present application is shown below. Fig.13 , the method may include the following steps:

[0142] S401. Back up all the data in the database to the backup space.

[0143] As a possible implementation method, the database system can perform a full backup of the data in the database at the initial backup time, and upload all current data in the database to the backup space.

[0144] It should be noted that the specific implementation method of this step can refer to the above S201 and will not be repeated here.

[0145] S402: After the full backup, incrementally back up the WAL files and metadata information to the backup space.

[0146] The WAL file is used to store write records, which include data and data timestamps, and the metadata information includes metadata and metadata timestamps.

[0147] It should be noted that the specific implementation of this step can refer to the above S202, which will not be repeated here. The difference is that after the full backup, the database system needs to incrementally back up the metadata information while incrementally backing up the WAL file.

[0148] The metadata in the data information refers to the data that defines the data structure. For example, in RocksDB, the metadata information may include at least one of the Manifest file, the Option file, and the Current file.

[0149] It should be introduced that when RocksDB performs compaction, local SST files are usually added and / or deleted. At this time, RocksDB can record the latest SST file version information (version) in the Manifest file. RocksDB has a column family (ColumnFamily) data structure. When a ColumnFamily is added or deleted, the corresponding operation is also directly recorded in the Manifest file, but this operation is usually not written to the WAL file. When a ColumnFamily is added, RocksDB will generate the latest Option file, which contains the configuration items of the newly added ColumnFamily. When a ColumnFamily is deleted, RocksDB will also regenerate the Option file, and the newly generated Option file will no longer contain the configuration items of the deleted ColumnFamily. The Current file is used to record the name of the Manifest file that RocksDB currently points to. When the current Manifest file is full and switched, the Current file will record the latest generated Manifest file name, and then the old Manifest can be deleted.

[0150] In some embodiments, in order to make the restored data structure the same as the original data structure in the database, the database system can add a timestamp to the above metadata information when backing up the data, and then back up the metadata information with the timestamp. When a database file (such as an SST file) in the database is modified, the database system can generate metadata information with the modification time according to the modification time.

[0151] Optionally, for the Manifest file, the database system can add a time parameter type in the encoding function of RocksDB. Furthermore, the database system can assign a value to the time type in the encoding function according to the modification time, and encode the Manifest file according to the assigned encoding function.

[0152] For example, RocksDB provides a pair of methods for encoding and decoding Manifest files, namely EncodeTo and DecodeFrom. The database system adds a data structure of type kSupportTimestamp in RocksDB, and the corresponding value is the modification time of the SST file. This type of data structure is placed in the initial stage of the EncodeTo function. When DecodeFrom is used, if it is determined to be of this type, it is assigned to timestamp, so that the modification time of the SST file can be obtained when restoring the data.

[0153] Optionally, for the Option file, the database system can add a configuration time option to the configuration options in the Option file. Furthermore, the database system assigns a value to the configuration time option according to the modification time, and adds the modification time to the Option file.

[0154] For example, the database system can add a new field option_timestamp when RocksDB constructs the content of the Options file (PersistRocksDBOptions) to store the time when RocksDB changes the Options file. Optionally, option_timestamp can be in the same directory as rocksdb_version and options_file_version in the Options file.

[0155] It can be understood that the Manifest file with a timestamp not only records the status changes of the database file in the database, but also records the modification time corresponding to the status changes of the database file. The Option file with a timestamp not only records the configuration options of the database file, but also records the configuration time corresponding to the configuration options.

[0156] like Fig.14 As shown in the figure, the data that the database system needs to back up may include SST files, WAL files, Manifest files, Option files, and Current files in RocksDB. Among them, the WAL files filled with Records can be first stored in the archive directory (i.e., the log directory to be backed up). The database system can upload the WAL files in the archive directory to the backup device and regularly clear the uploaded WAL files in the archive directory.

[0157] In some embodiments, when the database system performs data backup, at the initial backup time, the full metadata information in the database can also be backed up to the backup space, and the full metadata information includes all metadata information in the database at the initial backup time. After the full backup, the database system can incrementally back up the metadata information to the backup space.

[0158] For example, at the first backup time after the initial backup time, the database system may also send first incremental metadata information to the backup device, where the first incremental metadata information includes metadata information newly added to the metadata information in the database at the first backup time compared to the metadata information in the database at the initial backup time.

[0159] S403: When the data and metadata of the database need to be restored to the target time point, the full backup data, the target WAL file of the incremental backup, and the target metadata information of the incremental backup are obtained from the backup space.

[0160] Among them, the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, the target incremental backup is the most recent incremental backup of the WAL file after the target time point, and the target metadata information includes the metadata information of the target incremental backup and the metadata information of the incremental backup before the target incremental backup.

[0161] It should be noted that the specific implementation of this step can refer to the above S203, which will not be repeated here. The difference is that when the data and metadata of the database need to be restored to the target time point, the database system not only needs to obtain the target WAL file from the backup space, but also needs to obtain the target metadata information to support the subsequent metadata recovery process.

[0162] S404. On the basis of restoring the full backup data, by comparing the target time point with the timestamps of the data in the write records of the target WAL file, replay the write records in the target WAL file whose timestamps are before the target time point, so as to restore the database data to the target time point.

[0163] It should be noted that the specific implementation method of this step can refer to the above S204 and will not be repeated here.

[0164] S405: By comparing the target time point with the timestamps of the metadata in the target metadata information, the metadata in the target metadata information with the timestamp before the target time point is restored to restore the metadata of the database to the target time point.

[0165] In practical applications, the database system can decode the metadata in the target metadata information to obtain the timestamp corresponding to each metadata. Further, the database system compares the timestamp corresponding to each metadata with the target time point to recover the metadata in the target metadata information whose timestamp is before the target time point.

[0166] For example, Fig.15 As shown, after the database system obtains the target metadata information of the incremental backup from the backup space, it can traverse the Manifest files in the target metadata information and parse the timestamp (timestamp) corresponding to the kSupportTimestamp parameter in each Manifest file. When the timestamp of a Manifest file is greater than the pitr time, the database system stops traversing and restores the Manifest file before the pitr time.

[0167] Another exemplary example is Fig.16 As shown, after the database system obtains the target metadata information of the incremental backup from the backup space, it can traverse the Option files in the target metadata information and obtain the timestamp (option-timestamp) corresponding to each Option file. When the option-timestamp of an Option file is greater than the pitr time, the database system stops traversing and restores the Option files before the pitr time.

[0168] Similarly, after the database system obtains the target metadata information of the incremental backup from the backup space, it can traverse the Current files in the target metadata information and obtain the timestamp (current-timestamp) corresponding to each Current file. When the current-timestamp of a Current file is greater than the pitr time, the database system stops traversing and restores the Current file before the pitr time.

[0169] It is understandable that the database system adds a timestamp to the above metadata information when backing up data, and backs up the metadata information with a timestamp. When it is necessary to restore the data and metadata of the database to a target time point, the metadata before the target time point can be restored, and these metadata can make the restored data structure consistent with the original data structure in the database.

[0170] The present application also provides a data processing device, such as Fig.17 As shown, including:

[0171] A backup module, wherein the backup module is used for:

[0172] Back up the database data in full to the backup space; after the full backup, incrementally back up the WAL file to the backup space, wherein the WAL file is used to store write records, and the write record includes data and the timestamp of the data;

[0173] A recovery module, the recovery module is used to:

[0174] When the data of the database needs to be restored to the target time point, the full backup data and the target WAL file of the incremental backup are obtained from the backup space, the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, and the target incremental backup is the latest incremental backup after the target time point;

[0175] On the basis of restoring the full backup data, by comparing the target time point with the timestamps of the data in the write record of the target WAL file, the write record whose timestamp of the data in the target WAL file is before the target time point is played back to restore the data of the database to the target time point.

[0176] Among them, the backup module and the recovery module can be implemented by software or hardware. Exemplarily, the implementation of the backup module is introduced below by taking the backup module as an example. Similarly, the implementation of the recovery module can refer to the implementation of the backup module.

[0177] As an example of a software functional unit, a backup module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above computing instance may be one or more. For example, the backup module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0178] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0179] As an example of a hardware functional unit, the backup module may include at least one computing device, such as a server, etc. Alternatively, the backup module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0180] The multiple computing devices included in the backup module can be distributed in the same region or in different regions. The multiple computing devices included in the backup module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the backup module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0181] It should be noted that, in other embodiments, the backup module can be used to execute any step in the data processing method, and the recovery module can be used to execute any step in the data processing method. The steps that the backup module and the recovery module are responsible for implementing can be specified as needed, and the full functions of the data processing device are realized by implementing different steps in the data processing method through the backup module and the recovery module.

[0182] The present application also provides a data processing system, such as Fig.18 As shown, including:

[0183] A backup device, the backup device being used for:

[0184] Back up the database data in full to the backup space; after the full backup, incrementally back up the WAL file to the backup space, wherein the WAL file is used to store write records, and the write record includes data and the timestamp of the data;

[0185] A recovery device, the recovery device is used to:

[0186] When the data of the database needs to be restored to the target time point, the full backup data and the target WAL file of the incremental backup are obtained from the backup space, the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, and the target incremental backup is the latest incremental backup after the target time point;

[0187] On the basis of restoring the full backup data, by comparing the target time point with the timestamps of the data in the write record of the target WAL file, the write record whose timestamp of the data in the target WAL file is before the target time point is played back to restore the data of the database to the target time point.

[0188] The backup device and the recovery device can be implemented by software or hardware. As an example, the implementation of the backup device is described below. Similarly, the implementation of the recovery device can refer to the implementation of the backup device.

[0189] As an example of a software functional unit, a backup device may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the computing device may be one or more. For example, the backup device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same AZ or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Generally, a region may include multiple AZs.

[0190] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same VPC or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway must be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0191] As an example of a hardware functional unit, the backup device may include at least one computing device, such as a server, etc. Alternatively, the backup device may also be a device implemented using ASIC or PLD, etc. The PLD may be implemented using CPLD, FPGA, GAL or any combination thereof.

[0192] The multiple computing devices included in the backup device can be distributed in the same region or in different regions. The multiple computing devices included in the backup device can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the backup device can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0193] The present application also provides a computing device 100. Fig.19 As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0194] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.19 The bus 102 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0195] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0196] The memory 106 may include a volatile memory, such as a random access memory (RAM). The processor 104 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0197] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the backup module and the recovery module, thereby implementing the data processing method. That is, the memory 106 stores instructions for executing the data processing method.

[0198] Alternatively, the memory 106 stores executable codes, and the processor 104 executes the executable codes to respectively implement the functions of the backup device and the recovery device, thereby implementing the data processing method. That is, the memory 106 stores instructions for executing the data processing method.

[0199] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0200] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0201] like Fig. 20 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data processing method.

[0202] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the data processing method.

[0203] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the YY device. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more modules in the backup module and the recovery module.

[0204] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.21 A possible implementation is shown. Fig.21 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the backup module. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the recovery module.

[0205] Fig.21 The connection method between the computing device clusters shown may be based on the consideration that the data processing method provided in the present application needs to process a large amount of data, and therefore it is considered that the functions implemented by the backup module and the recovery module are handed over to the computing device 100B for execution.

[0206] It should be understood that Fig.21 The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0207] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig. 20 and Fig.21 The connection mode of the computing device cluster is different in that the memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data processing method.

[0208] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the data processing method.

[0209] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions for executing partial functions of the data processing system. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more of the backup device and the recovery device.

[0210] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method of the embodiment of the present application.

[0211] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method of the embodiment of the present application.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to a database system, wherein the database system uses a write-ahead log (WAL) to record operations on the database, and the method comprises: Backing up all the data in the database to the backup space; After the full backup, incrementally back up the WAL file to the backup space, wherein the WAL file is used to store write records, and the write records include data and a timestamp of the data; When the data of the database needs to be restored to the target time point, the full backup data and the target WAL file of the incremental backup are obtained from the backup space, the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, and the target incremental backup is the latest incremental backup after the target time point; On the basis of restoring the full backup data, by comparing the target time point with the timestamps of the data in the write record of the target WAL file, the write record whose timestamp of the data in the target WAL file is before the target time point is played back to restore the data of the database to the target time point.

2. The method according to claim 1, characterized in that: The method further comprises: Receive write requests from clients; In response to the write request, the data corresponding to the write request and the timestamp of the data corresponding to the write request are written into a WAL file to obtain a write record corresponding to the write request.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Store the full WAL file in the directory to be backed up; The incrementally backing up the WAL files to the backup space includes: incrementally backing up the WAL files in the directory to be backed up to the backup space.

4. The method according to claim 3, characterized in that The method further comprises: Delete the WAL files that have been incrementally backed up to the backup space.

5. A data processing method, characterized in that: The method is applied to a database system, wherein the database system uses a write-ahead log (WAL) to record operations on the database, and the method comprises: Backing up all the data in the database to the backup space; After the full backup, incrementally back up the WAL file to the backup space, wherein the WAL file is used to store write records, and the write records include data and a timestamp of the data; After the full backup, the data of the database is incrementally backed up to the backup space, wherein at least one incremental backup of the WAL file is performed between every two incremental backups of the data; When it is necessary to restore the data of the database to the target time point, the data of the full backup, the target data of the incremental backup, and the target WAL file of the incremental backup are obtained from the backup space; wherein the target data includes the data of the first incremental backup and the data of the incremental backup before the first incremental backup, the first incremental backup is the latest incremental backup of the data before the target time point, the target WAL file includes the WAL file of the second incremental backup, the WAL file before the second incremental backup and after the first incremental backup, and the second incremental backup is the latest incremental backup of the WAL file after the target time point; On the basis of restoring the target data of the full backup and the incremental backup, by comparing the target time point and the timestamp of the data in the write record of the target WAL file, the write record whose timestamp of the data in the target WAL file is before the target time point is played back to restore the data of the database to the target time point.

6. A data processing method, characterized in that: The method is applied to a database system, wherein the database system uses a write-ahead log (WAL) to record operations on the database, and the method comprises: Backing up all the data in the database to the backup space; After the full backup, incrementally back up the WAL file and metadata information to the backup space, wherein the WAL file is used to store write records, the write records include data and the timestamp of the data, and the metadata information includes metadata and the timestamp of the metadata; When it is necessary to restore the data and metadata of the database to the target time point, the data of the full backup, the target WAL file of the incremental backup, and the target metadata information of the incremental backup are obtained from the backup space, wherein the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, the target incremental backup is the latest incremental backup of the WAL file after the target time point, and the target metadata information includes the metadata information of the target incremental backup and the metadata information of the incremental backup before the target incremental backup; On the basis of restoring the full backup data, by comparing the target time point with the timestamps of the data in the write records of the target WAL file, replaying the write records of the data in the target WAL file whose timestamps are before the target time point, so as to restore the data of the database to the target time point; By comparing the target time point with the timestamps of the metadata in the target metadata information, the metadata in the target metadata information with a timestamp before the target time point is restored, so as to restore the metadata of the database to the target time point.

7. A data processing system, characterized in that: The system comprises: A backup device, the backup device being used for: Back up the database data in full to the backup space; after the full backup, incrementally back up the WAL file to the backup space, wherein the WAL file is used to store write records, and the write record includes data and the timestamp of the data; A recovery device, the recovery device is used to: When the data of the database needs to be restored to the target time point, the full backup data and the target WAL file of the incremental backup are obtained from the backup space, the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, and the target incremental backup is the latest incremental backup after the target time point; On the basis of restoring the full backup data, by comparing the target time point with the timestamps of the data in the write record of the target WAL file, the write record whose timestamp of the data in the target WAL file is before the target time point is played back to restore the data of the database to the target time point.

8. The system according to claim 7, characterized in that The system further comprises: A receiving device, the receiving device is used to receive a write request from a client; A response device is used to respond to the write request and write the data corresponding to the write request and the timestamp of the data corresponding to the write request into a WAL file to obtain a write record corresponding to the write request.

9. The system according to claim 7 or 8, characterized in that: The backup device is also used to store the full WAL file in the directory to be backed up; The backup device is used to incrementally back up the WAL file to the backup space, including: The backup device is used for incrementally backing up the WAL files in the directory to be backed up to the backup space.

10. The system according to claim 9, characterized in that The backup device is also used to delete the WAL files that have been incrementally backed up to the storage space.

11. A data processing system, characterized in that: The system comprises: A backup device, the backup device being used for: Backing up the data of the database in full to the backup space; after the full backup, incrementally backing up the WAL file to the backup space, wherein the WAL file is used to store write records, and the write records include data and timestamps of the data; after the full backup, incrementally backing up the data of the database to the backup space, wherein at least one incremental backup of the WAL file is performed between every two incremental backups of the data; A recovery device, the recovery device is used to: When it is necessary to restore the data of the database to the target time point, the data of the full backup, the target data of the incremental backup, and the target WAL file of the incremental backup are obtained from the backup space; wherein the target data includes the data of the first incremental backup and the data of the incremental backup before the first incremental backup, the first incremental backup is the latest incremental backup of the data before the target time point, the target WAL file includes the WAL file of the second incremental backup, the WAL file before the second incremental backup and after the first incremental backup, and the second incremental backup is the latest incremental backup of the WAL file after the target time point; On the basis of restoring the target data of the full backup and the incremental backup, by comparing the target time point and the timestamp of the data in the write record of the target WAL file, the write record whose timestamp of the data in the target WAL file is before the target time point is played back to restore the data of the database to the target time point.

12. A data processing system, characterized in that: The system comprises: A backup device, the backup device being used for: Backing up the data of the database in full to the backup space; after the full backup, incrementally backing up the WAL file and metadata information to the backup space, wherein the WAL file is used to store write records, the write record includes data and a timestamp of the data, and the metadata information includes metadata and a timestamp of the metadata; A recovery device, the recovery device is used to: When it is necessary to restore the data and metadata of the database to the target time point, the data of the full backup, the target WAL file of the incremental backup, and the target metadata information of the incremental backup are obtained from the backup space, wherein the target WAL file includes the WAL file of the target incremental backup and the WAL file of the incremental backup before the target incremental backup, the target incremental backup is the latest incremental backup of the WAL file after the target time point, and the target metadata information includes the metadata information of the target incremental backup and the metadata information of the incremental backup before the target incremental backup; On the basis of restoring the full backup data, by comparing the target time point with the timestamps of the data in the write records of the target WAL file, replaying the write records of the data in the target WAL file whose timestamps are before the target time point, so as to restore the data of the database to the target time point; By comparing the target time point with the timestamps of the metadata in the target metadata information, the metadata in the target metadata information with a timestamp before the target time point is restored, so as to restore the metadata of the database to the target time point.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the data processing method according to any one of claims 1 to 6.

14. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the data processing method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the data processing method according to any one of claims 1 to 6.

Images