Data backup recovery method, device, equipment and product for cloud data warehouse

Abstract

The invention discloses a data backup recovery method, device, equipment and product for a cloud data warehouse, and the method comprises the steps: backing up metadata related to table data and the table data into an object storage medium at the same time according to the range of data needing to be backed up; according to the range of the data needing to be backed up, the remaining metadata except the metadata related to the table data are backed up into the object storage medium; and when the data is recovered, according to the snapshot and the directory of the backup data, the backup data is recovered and written back to the original storage position. The invention discloses a data backup and recovery method, device, equipment and product for a cloud data warehouse, realizes data backup and recovery of the cloud data warehouse, improves the remote disaster recovery capability of the data warehouse, and has the characteristics of complete data backup and high reliability.

Description

Technical Field

[0001] The present invention belongs to the field of databases, and in particular relates to a data backup and recovery method, device, equipment and product for a cloud data warehouse. Background Art

[0002] Under existing technical conditions, cloud data warehouse products have been widely used due to their excellent performance. However, in some cloud data warehouse products, according to the data storage method they set, metadata is stored in the distributed database FDB, and table data is stored in the object storage medium. Direct data backup and recovery cannot be achieved, so they do not have complete off-site disaster recovery capabilities, which reduces the reliability and performance of the database. Summary of the invention

[0003] In view of this, the present invention aims to overcome the defects in the prior art and proposes a data backup and recovery method, device, equipment and product for a cloud data warehouse.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows: In a first aspect, the present invention discloses a data backup and recovery method for a cloud data warehouse, comprising: according to the scope of data to be backed up, metadata related to table data and the table data are backed up simultaneously to an object storage medium, wherein the metadata related to the table data is a metadata file corresponding to the table data, a metadata file index, a metadata file index sequence number, and table structure information; Based on the scope of data to be backed up, the remaining metadata other than metadata related to the table data is backed up to the object storage medium; During data recovery, the backed-up metadata is restored and written back to the original storage location based on the snapshot and directory of the metadata backed up to the object storage medium; the backed-up table data is restored and written back to the original storage location based on the snapshot and directory of the table data backed up to the object storage medium.

[0005] In one embodiment of the present invention, the range of data that needs to be backed up is determined according to the set data backup granularity.

[0006] In one embodiment of the present invention, based on the scope of data to be backed up, metadata related to table data and table data are backed up simultaneously to the object storage medium, including storing the metadata to be backed up in the distributed database FDB and storing the table data to be backed up in the object storage medium.

[0007] In one embodiment of the present invention, metadata related to table data and the table data are backed up to an object storage medium at the same time, including: parsing all metadata file keys according to metadata file indexes, and obtaining corresponding table data according to the metadata file keys.

[0008] In one embodiment of the present invention, during data recovery, based on the snapshot and directory of the metadata backed up in the object storage medium, before the backed-up metadata is restored and written back to the original storage location, it also includes: before data recovery, determining that there are no objects in the backup object list in the cluster, if the objects in the backup object list exist in the cluster, then deleting them before data recovery.

[0009] In one embodiment of the present invention, table data is backed up to the object storage medium using a full backup or an incremental backup.

[0010] In one embodiment of the present invention, metadata related to table data and table data are backed up simultaneously to the object storage medium according to the scope of data to be backed up, including: metadata related to table data and table data are both the latest version data and correspond to each other.

[0011] In a second aspect, the present invention discloses a data backup and recovery device for a cloud data warehouse, the device comprising: A first backup module is used to back up metadata related to the table data and the table data to the object storage medium at the same time according to the scope of data that needs to be backed up, wherein the metadata related to the table data is a metadata file corresponding to the table data, a metadata file index, a metadata file index sequence number, and table structure information; The second backup module is used to back up the remaining metadata other than the metadata related to the table data to the object storage medium according to the range of data that needs to be backed up; The data recovery module is used to restore the backed-up metadata and write it back to the original storage location based on the snapshot and directory of the metadata backed up in the object storage medium during data recovery; and to restore the backed-up table data and write it back to the original storage location based on the snapshot and directory of the table data backed up in the object storage medium.

[0012] In a third aspect, the present invention discloses an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0013] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0014] Compared with the prior art, the present invention has the following advantages: The present invention discloses a data backup and recovery method, device, equipment and product for a cloud data warehouse, including: according to the scope of data to be backed up, metadata related to table data and table data are backed up to an object storage medium at the same time; according to the scope of data to be backed up, the remaining metadata other than metadata related to table data is backed up to an object storage medium; when data is restored, according to the snapshot and directory of the metadata backed up to the object storage medium, the backed-up metadata is restored and written back to the original storage location; according to the snapshot and directory of the table data backed up to the object storage medium, the backed-up table data is restored and written back to the original storage location. The present invention discloses a data backup and recovery method, device, equipment and product for a cloud data warehouse, which realizes data backup and recovery of a cloud data warehouse, improves the off-site disaster recovery capability of the data warehouse, and has the characteristics of complete data backup and high reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0016] In the attached picture: Figure 1 A schematic diagram of an application scenario of a data backup and recovery method for a cloud data warehouse according to an embodiment of the present invention; Figure 2 A schematic diagram of a data backup and recovery method for a cloud data warehouse according to an embodiment of the present invention; Figure 3 A schematic diagram of determining table data for a data backup and recovery method for a cloud data warehouse according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a data backup and recovery device for a cloud data warehouse according to an embodiment of the present invention; Figure 5 The present invention is a schematic diagram of an electronic device for a data backup and recovery method for a cloud data warehouse according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0018] In the description of the present invention, it should be further explained that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0019] The application scenarios of the data backup and recovery method, device, equipment and product for cloud data warehouse disclosed in the present invention are as follows: Figure 1 As shown, under existing technical conditions, in some cloud data warehouse products, according to their set data storage method, metadata is stored in the distributed database FDB, and table data is stored in the object storage medium. Data backup and recovery cannot be simply and directly performed, so it does not have a complete off-site disaster recovery capability, which reduces the reliability and performance of the database.

[0020] The present invention discloses a data backup and recovery method, device, equipment and product for a cloud data warehouse. According to the scope of data to be backed up, metadata related to table data and table data are backed up to an object storage medium at the same time; according to the scope of data to be backed up, the remaining metadata other than metadata related to table data is backed up to an object storage medium; further, when data is restored, according to the snapshot and directory of the metadata backed up to the object storage medium, the backed-up metadata is restored and written back to the original storage location; according to the snapshot and directory of the table data backed up to the object storage medium, the backed-up table data is restored and written back to the original storage location. The present invention realizes data backup and recovery of a cloud data warehouse, improves the off-site disaster recovery capability of a data warehouse, and has the characteristics of complete data backup and high reliability.

[0021] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] In one embodiment of the present invention, Figure 2 As shown, a data backup and recovery method for a cloud data warehouse includes: Step S201, according to the scope of data to be backed up, metadata related to table data and table data are backed up to the object storage medium at the same time, wherein the metadata related to the table data is the metadata file corresponding to the table data, the metadata file index, the metadata file index sequence number and the table structure information; In this embodiment, metadata files, metadata file indexes, metadata file index sequence numbers, and table structure information are stored in a non-mstable format, and the pfile module provides an interface for reading and writing.

[0023] In this embodiment, if Figure 1 As shown, before data backup, the metadata to be backed up is stored in the distributed database FDB, and the table data to be backed up is stored in the object storage medium. Exemplarily, the object storage medium can be the object storage service S3 or the Hadoop distributed file system HDFS.

[0024] In this embodiment, data backup is performed at a set time interval, and an administrator who performs data backup can create a backup transaction based on a consistent view to ensure data integrity.

[0025] In this embodiment, the range of data that needs to be backed up is determined according to the set data backup granularity.

[0026] Exemplarily, the data backup granularity may be database backup or data table backup, and the corresponding range of data that needs to be backed up is the database or the data table.

[0027] In this embodiment, metadata related to table data and table data are backed up to the object storage medium at the same time according to the scope of data to be backed up, including: metadata related to table data and table data are both the latest version data and correspond to each other.

[0028] Because there are multiple data versions in the cloud data warehouse, some early data versions have become junk data, which does not need to be backed up. Therefore, the metadata related to the table data and the latest version of the table data are selected for backup, and the metadata related to the table data and the table data correspond to each other.

[0029] Step S202: Backing up the remaining metadata other than the metadata related to the table data to the object storage medium according to the scope of the data to be backed up; Exemplarily, the remaining metadata may be database name, view, function, stored procedure, etc.

[0030] The remaining metadata is stored in mstable format. The pfile module provides an interface for reading and writing, with each object as the reading unit, reading based on the mstable primary key, and reading and writing in parallel or asynchronous mode.

[0031] Step S203, when restoring data, based on the snapshot and directory of the metadata backed up in the object storage medium, the backed-up metadata is restored and written back to the original storage location; based on the snapshot and directory of the table data backed up in the object storage medium, the backed-up table data is restored and written back to the original storage location.

[0032] In this embodiment, the snapshot is used to determine the data that needs to be restored at the current moment, and the directory is used to determine the storage location of the data that needs to be restored.

[0033] In this embodiment, data backup is performed at set time intervals, so there are multiple backup points. Based on the snapshots and directories of the metadata backed up to the object storage medium, the backed up metadata is restored and written back to the original storage location, that is, it is restored to the last backup point.

[0034] In another embodiment of the present invention, Figure 3As shown, metadata related to table data and table data are backed up to the object storage medium at the same time, including: parsing all metadata file keys according to the metadata file index, and obtaining corresponding table data according to the metadata file key.

[0035] In another embodiment of the present invention, during data recovery, based on the snapshot and directory of the metadata backed up in the object storage medium, before the backed up metadata is restored and written back to the original storage location, it also includes: before data recovery, determining that there are no objects in the backup object list in the cluster, if the objects in the backup object list exist in the cluster, then deleting them before data recovery.

[0036] In this embodiment, since the backup granularity requirement is the same in each backup cycle, only one backup object list needs to be maintained in each backup cycle.

[0037] In another embodiment of the present invention, the table data is backed up to the object storage medium using a full backup or an incremental backup.

[0038] In this embodiment, when the table data is backed up to the object storage medium, the historical backup is checked. If it is a full backup, it is checked whether the directory to be written already exists. If it does, an error is reported and the process is exited. If it is an incremental backup, it is checked whether there is a valid historical table-level backup. If not, an error is reported and the process is exited. If there is junk table-level data, the user is prompted to clear it.

[0039] In this embodiment, when performing incremental backup, each backup operation also backs up the current version snapshot, and in the next incremental backup, the snapshot generated at the current backup time is compared with the snapshot backed up in the previous backup point to back up the differential version.

[0040] To ensure the consistency of data backup, the method is as follows: First, the database objects seen when starting the backup are not allowed to be deleted or modified during the backup process. Newly added database objects during the backup process are not within the backup scope. This is achieved by adding a cluster lock and generating a backup list object at the start of the backup.

[0041] Second, all data that can be seen when the backup starts needs to be backed up. The data that can be seen when the backup starts is determined based on the transaction snapshot, thereby determining the backup scope. At the same time, it is necessary to ensure that the data that needs to be backed up cannot be cleaned up by garbage collection during the backup process.

[0042] Third, various globally unique ID information such as database ID, data table ID, transaction ID, etc., remain unchanged during recovery.

[0043] like Figure 4 As shown, the present invention also discloses a data backup and recovery device, comprising: The first backup module 401 is used to back up metadata related to the table data and the table data to the object storage medium at the same time according to the scope of data to be backed up, wherein the metadata related to the table data is the metadata file corresponding to the table data, the metadata file index, the metadata file index sequence number and the table structure information; The second backup module 402 is used to back up the remaining metadata other than the metadata related to the table data to the object storage medium according to the range of data to be backed up; The data recovery module 403 is used to restore the backed-up metadata and write it back to the original storage location based on the snapshot and directory of the metadata backed up in the object storage medium during data recovery; and to restore the backed-up table data and write it back to the original storage location based on the snapshot and directory of the table data backed up in the object storage medium.

[0044] The present invention also discloses an electronic device, such as Figure 5 As shown, an embodiment is disclosed, which is a block diagram of an electronic device suitable for the above-mentioned data backup and recovery method for a cloud data warehouse.

[0045] The electronic device 50 of this embodiment includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the ROM 502 or the program loaded from the storage part 508 to the RAM 503. The processor 501 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a dedicated microprocessor, etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present invention.

[0046] In RAM503, various programs and data required for the operation of electronic device 50 are stored. Processor 501, ROM502 and RAM503 are connected to each other via bus 504, and processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM502 and / or RAM503. It should be noted that the program can also be stored in one or more memories other than ROM502 and RAM503, and processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0047] According to an embodiment of the present invention, the electronic device 50 may further include an I / O interface 505, which is also connected to the bus 504. The electronic device 50 may further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube, a liquid crystal display, and a speaker; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 5010 is also connected to the I / O interface 505 as needed. A removable medium 5011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 5010 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0048] Embodiments of the present invention also include a computer program product.

[0049] The computer program product includes a computer program, which contains program codes for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program codes are used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0050] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium. The program code included in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0051] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written by any combination of one or more programming languages, and specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages. Programming languages ​​include but are not limited to programming languages ​​such as Java, C++, python, C language or similar. The program code can be executed completely on the user computing device, partially on the user device, partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network or a wide area network, or can be connected to an external computing device.

[0052] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teaching of the present invention, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0053] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination. The scope of the present invention is limited by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data backup and recovery method for a cloud data warehouse, characterized in that: include: According to the scope of data to be backed up, metadata related to the table data and the table data are backed up simultaneously to the object storage medium, wherein the metadata related to the table data is a metadata file corresponding to the table data, a metadata file index, a metadata file index sequence number, and table structure information; Backing up the remaining metadata other than the metadata related to the table data into the object storage medium according to the range of the data to be backed up; During data recovery, the backed-up metadata is restored and written back to the original storage location based on the snapshot and directory of the metadata backed up in the object storage medium; the backed-up table data is restored and written back to the original storage location based on the snapshot and directory of the table data backed up in the object storage medium.

2. A data backup and recovery method for a cloud data warehouse according to claim 1, characterized in that: Determine the scope of data that needs to be backed up based on the set data backup granularity.

3. A data backup and recovery method for a cloud data warehouse according to claim 1, characterized in that: The method of backing up the metadata related to the table data and the table data to the object storage medium at the same time according to the scope of the data to be backed up also includes storing the metadata to be backed up in the distributed database FDB and storing the table data to be backed up in the object storage medium.

4. A data backup and recovery method for a cloud data warehouse according to claim 1, characterized in that: The step of simultaneously backing up the metadata related to the table data and the table data to the object storage medium includes: parsing all metadata file keys according to the metadata file index, and acquiring the corresponding table data according to the metadata file key.

5. The data backup and recovery method for a cloud data warehouse according to claim 1, characterized in that: During data recovery, the backed-up metadata is restored and written back to the original storage location based on the snapshot and directory of the metadata backed up in the object storage medium. It also includes: before data recovery, determining that there are no objects in the backup object list in the cluster. If the objects in the backup object list exist in the cluster, they are deleted before data recovery.

6. A data backup and recovery method for a cloud data warehouse according to claim 1, characterized in that: The table data is backed up to the object storage medium using a full backup or an incremental backup.

7. A data backup and recovery method for a cloud data warehouse according to claim 1, characterized in that: The step of backing up metadata related to the table data and the table data simultaneously to the object storage medium according to the scope of the data to be backed up includes: the metadata related to the table data and the table data are both the latest version data and correspond to each other.

8. A data backup and recovery device for a cloud data warehouse, characterized in that: The device comprises: A first backup module is used to back up metadata related to table data and the table data to an object storage medium at the same time according to the scope of data that needs to be backed up, wherein the metadata related to the table data is a metadata file corresponding to the table data, a metadata file index, a metadata file index sequence number, and table structure information; A second backup module, configured to back up the remaining metadata other than the metadata related to the table data into the object storage medium according to the range of the data to be backed up; The data recovery module is used to restore the backed-up metadata and write it back to the original storage location according to the snapshot and directory of the metadata backed up in the object storage medium during data recovery; and to restore the backed-up table data and write it back to the original storage location according to the snapshot and directory of the table data backed up in the object storage medium.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to perform the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images