Distributed Backup Method and System for Open MPP Database Based on Data Warehouse

Abstract

The present invention provides a distributed backup method and system based on an open MPP database of a data warehouse, including: Step S1: Classify the backup tables in the database based on business processing logic; Step S2: Subdivide the classified tables into different ordered time granularities through time granularity; Step S3: According to the classification of the tables and the time granularity, determine the backup methods as incremental backup, full backup, and overall backup; Step S4: Perform distributed backup, and each write bandwidth is exclusively used through parallel writing. The backup efficiency of the present invention is greatly improved; different backup strategies are designed according to different database table types. While ensuring the integrity of the backup data, the actual backup is performed according to the changed data. The distributed backup mode solves the network bottleneck and improves the backup concurrent writing ability. Therefore, the backup time is greatly reduced to meet the requirements of large data volume backup.

Description

Technical Field

[0001] The present invention relates to the field of computers, and specifically, to a distributed backup method and system based on an open MPP database of a data warehouse. Background Art

[0002] With the advent of the era of massive data processing, the requirements for distributed data processing capabilities are getting higher and higher. Distributed databases will gradually replace single-machine databases, and the original single-machine backup to related devices such as EMC has a single-point bottleneck.

[0003] Patent document CN111651302A (application number: CN202010618794.3) discloses a distributed database backup method, device and system. The system includes: a master node unit for generating a plurality of node database backup commands according to pre-set database sub-libraries and sub-tables; each node database backup command is used to indicate the data sub-tables that need to be backed up in the corresponding node database; each sub-node backup module is used to back up the data sub-tables that need to be backed up in the corresponding node database according to the corresponding node database backup command, generate backup files, and copy the backup files to a shared storage server. However, this invention does not design different backup strategies according to different database table types. Summary of the Invention

[0004] Aiming at the defects in the prior art, the purpose of the present invention is to provide a distributed backup method and system based on an open MPP database of a data warehouse.

[0005] According to a distributed backup method based on an open MPP database of a data warehouse provided by the present invention, it includes:

[0006] Step S1: Classify the backup tables in the database based on business processing logic;

[0007] Step S2: Subdivide the classified tables into different ordered time granularities through time granularity;

[0008] Step S3: According to the classification of the tables and time granularity, determine the backup methods as incremental backup, full backup and overall backup;

[0009] Step S4: Perform distributed backup, and the single write bandwidth is exclusive through parallel writing.

[0010] Preferably, in the step S1:

[0011] Classify the backup tables in the database based on business processing logic according to data management, and the obtained table classifications include: F1 algorithm, F2 algorithm, I algorithm and F5 algorithm;

[0012] Among them, the F1 algorithm is for full insertion. When each job runs, all data in the current data table is deleted, and then the operation result of the current business date is retained without audit fields.

[0013] The F2 algorithm is for incremental update. Records in the data table are updated according to the primary key. This type of data processing job has the characteristics of storing historical data and retaining its latest status except for the primary key, and has audit fields.

[0014] The I algorithm is for direct insertion. Records of selected fields are appended to the end of the data table and have audit fields.

[0015] The F5 algorithm is for historical zipper. When there is an update to the data except for the primary key, the end_date of the original record is updated to the day before the current business date; then a new record is added with the start_date set to the current business date, and the status values of the corresponding fields are recorded, and the end_date field is set.

[0016] Preferably, in the step S2:

[0017] The classification of the table is subdivided into different ordered time granularities according to the actual situation through time granularity, namely: daily, weekly, and monthly.

[0018] Among them, when the table type is F1, F2, I, and F5, the daily frequency is performed; when the table type is I and F5, the weekly frequency backup is performed; when the table type is F1, F2, I, and F5, the monthly frequency backup is performed.

[0019] Preferably, in the step S3:

[0020] According to the determined table classification and time granularity, it is determined as incremental backup, full backup, and overall backup through the backup method.

[0021] The data volume of the data processed by the F1 and F2 algorithms is small, and incremental backup is performed daily; the data volume of the data processed by the I and F5 algorithms is large. On the basis of performing incremental backup daily, a full backup is performed once a week; all data in the database is subject to an overall backup once a month.

[0022] For the classification of F1 and F2, the backup frequency is daily, and the backup method is determined as incremental backup; incremental backup is after a full backup or the previous incremental backup. For each subsequent backup, only the data that has increased and been modified compared with the previous one needs to be backed up, and it is restored through the most recent full backup and all incremental backups.

[0023] For the tables classified as I and F5, with a backup frequency of once a week and the backup method determined as full backup, a full backup makes a complete copy of all data at a certain point in time and only requires the last full backup to be restored;

[0024] For the tables classified as F1, F2, I and F5, with a backup frequency of once a month and the backup method determined as overall backup, an overall backup makes a full backup of all data in the database at a certain point in time.

[0025] Preferably, in the step S4:

[0026] Perform distributed backup. Through the parallel writing method, each writing bandwidth is exclusive; the number of MPP cluster nodes and backup nodes is in a preset ratio; mount the backup node file directory on the cluster node through the NFS service; as the number of MPP nodes increases, increase the number of backup servers according to the preset ratio.

[0027] According to a distributed backup system based on an open MPP database of a data warehouse provided by the present invention, it includes:

[0028] Module M1: Classify the backup tables in the database based on the business processing logic;

[0029] Module M2: Subdivide the classified tables into different ordered time granularities through the time granularity;

[0030] Module M3: According to the classification of the tables and the time granularity, determine the backup methods as incremental backup, full backup and overall backup through judgment;

[0031] Module M4: Perform distributed backup, and each writing bandwidth is exclusive through the parallel writing method.

[0032] Preferably, in the module M1:

[0033] Classify the backup tables in the database based on the business processing logic according to data management, and the obtained table classifications include: F1 algorithm, F2 algorithm, I algorithm and F5 algorithm;

[0034] Among them, the F1 algorithm is full insert. When each job runs, all data in the current data table is deleted, and then the operation result of the current business date is retained, without audit fields;

[0035] The F2 algorithm is incremental update. Update the records in the data table according to the primary key. This type of data processing job has the characteristics of storing historical data and retaining its latest state except the primary key, and has audit fields;

[0036] The I algorithm is direct insert. Append the records of the selected fields to the end of the data table, and has audit fields;

[0037] The F5 algorithm is a historical zipper. When there are updates to data other than the primary key, the end_date of the original record is updated to the day before the current business date. Subsequently, a new record is added with the start_date set to the current business date, the status values of the corresponding fields are recorded, and the end_date field is set.

[0038] Preferably, in the module M2:

[0039] The classification of the table is subdivided into different ordered time granularities according to the actual situation through the time granularity, namely: daily, weekly, and monthly.

[0040] Among them, in the case of table types F1, F2, I, and F5, the daily frequency is performed; in the case of table types I and F5, the weekly frequency backup is performed; in the case of table types F1, F2, I, and F5, the monthly frequency backup is performed.

[0041] Preferably, in the module M3:

[0042] According to the determined table classification and time granularity, it is determined as incremental backup, full backup, and overall backup through the backup method;

[0043] The data volume of the data processed by the F1 and F2 algorithms is small, and incremental backup is performed daily; the data volume of the data processed by the I and F5 algorithms is large. On the basis of performing incremental backup daily, a full backup is performed once a week; all the data in the database is subjected to an overall backup once a month;

[0044] For the classification of F1 and F2, the backup frequency is daily, and the backup method is determined as incremental backup; incremental backup is after a full backup or the previous incremental backup. For each subsequent backup, only the data that has increased and been modified compared to the previous one needs to be backed up, and it is restored through the most recent full backup and all incremental backups;

[0045] For the classification of the table as I and F5, the backup frequency is weekly, and the backup method is determined as full backup. Full backup is a complete copy of all the data at a certain point in time, and it can be restored only with the previous full backup;

[0046] For the classification of the table as F1, F2, I, and F5, the backup frequency is monthly, and the backup method is determined as overall backup. Overall backup is to perform a full backup of all the data in the database at a certain point in time.

[0047] Preferably, in the module M4:

[0048] Perform distributed backup. Through the method of parallel writing, each write bandwidth is exclusive. The number of MPP cluster nodes and backup nodes is in a preset ratio. Mount the backup node file directory on the cluster nodes through the NFS service. As the number of MPP nodes increases, the number of backup servers is increased according to the preset ratio.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. The backup efficiency of the present invention is greatly improved; different backup strategies are designed according to different database table types. While ensuring the integrity of the backup data, the actual backup is performed according to the changed data. The distributed backup mode solves the network bottleneck and improves the backup concurrent writing ability. Therefore, the backup time is greatly reduced to meet the needs of large-scale data backup.

[0051] 2. The backup devices of the present invention are inexpensive; cheap PC servers are used to replace professional backup devices, and the cost is reduced by at least more than 50%. The backup deployment mode is more flexible and simple.

[0052] 3. The present invention supports horizontal expansion. The present invention can arbitrarily increase the backup device incrementally according to needs without affecting the overall backup efficiency. Existing backups using professional backup devices generally do not support horizontal storage and network horizontal expansion. Once the backup device needs to be expanded and upgraded, it may be necessary to purchase a new independent backup device, and it is not possible to incrementally add devices based on the original devices, resulting in waste of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] By reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings, other features, objects, and advantages of the present invention will become more apparent:

[0054] Figure 1 It is a deployment diagram of the backup solution;

[0055] Figure 2 It is a distributed backup mechanism diagram. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all belong to the protection scope of the present invention.

[0057] Embodiment 1;

[0058] According to a distributed backup method for an open MPP database based on a data warehouse provided by the present invention, as Figure 1 - Figure 2 shown, it includes:

[0059] Step S1: Classify the backup tables in the database based on the business processing logic;

[0060] Specifically, in the step S1:

[0061] Classify the backup tables in the database based on the business processing logic according to data management, and the classifications of the tables obtained include: F1 algorithm, F2 algorithm, I algorithm, and F5 algorithm;

[0062] Among them, the F1 algorithm is full - volume insertion. When each job runs, all data in the current data table is deleted, and then the operation result of the current business date is retained, without audit fields;

[0063] The F2 algorithm is incremental update. The records in the data table are updated according to the primary key. This type of data - processing job has the characteristics of storing historical data and retaining its latest state except for the primary key, and has audit fields;

[0064] The I algorithm is direct insertion. The records of the selected fields are appended to the end of the data table, and have audit fields;

[0065] The F5 algorithm is historical zipper. When there is an update to the data except for the primary key, the end_date of the original record is updated to the day before the current business date; then a new record is added with the start_date set to the current business date, and the status values of the corresponding fields are recorded, and the end_date field is set.

[0066] Step S2: Sub - divide the classified tables into different ordered time granularities through time granularity;

[0067] Specifically, in the step S2:

[0068] Sub - divide the classification of the tables into different ordered time granularities according to the actual situation through time granularity, which are: daily, weekly, and monthly;

[0069] Among them, in the case of table types F1, F2, I, and F5, the daily frequency is performed; in the case of table types I and F5, the weekly frequency backup is performed; in the case of table types F1, F2, I, and F5, the monthly frequency backup is performed.

[0070] Step S3: Determine the backup methods as incremental backup, full - volume backup, and overall backup according to the classification of the tables and the time granularity;

[0071] Specifically, in the step S3:

[0072] Determine the backup methods as incremental backup, full - volume backup, and overall backup according to the determined table classification and time granularity;

[0073] The data processed by algorithms F1 and F2 has a relatively small amount, and incremental backups are performed daily; the data processed by algorithms I and F5 has a relatively large amount, and in addition to daily incremental backups, full backups are performed once a week; all the data in the database is overall backed up once a month.

[0074] For the classification of F1 and F2, the backup frequency is daily, and the backup method is determined as incremental backup; incremental backup is after a full backup or the previous incremental backup, and for each subsequent backup, only the data that has increased and been modified compared to the previous one needs to be backed up, and recovery is through the most recent full backup and all incremental backups.

[0075] For the classification of I and F5 for the table, the backup frequency is weekly, and the backup method is determined as full backup. A full backup is a complete copy of all the data at a certain point in time, and recovery only requires the previous full backup.

[0076] For the classification of F1, F2, I, and F5 for the table, the backup frequency is monthly, and the backup method is determined as overall backup. An overall backup is a full backup of all the data in the database at a certain point in time.

[0077] Step S4: Perform distributed backup, and the individual write bandwidth is exclusive through the parallel write method.

[0078] Specifically, in the said step S4:

[0079] Perform distributed backup, through the parallel write method, and the individual write bandwidth is exclusive; the number of MPP cluster nodes and the backup nodes are in a preset ratio; the backup node file directory is mounted on the cluster nodes through the NFS service; as the number of MPP nodes increases, the number of backup servers increases according to the preset ratio.

[0080] Embodiment 2:

[0081] Embodiment 2 is a preferred example of Embodiment 1 to more specifically illustrate the present invention.

[0082] The present invention also provides a distributed backup system based on a data warehouse open-type MPP database. The distributed backup system based on the data warehouse open-type MPP database can be implemented by executing the process steps of the distributed backup method based on the data warehouse open-type MPP database. That is, those skilled in the art can understand the distributed backup method based on the data warehouse open-type MPP database as the preferred implementation manner of the distributed backup system based on the data warehouse open-type MPP database.

[0083] According to a distributed backup system based on a data warehouse open-type MPP database provided by the present invention, it includes:

[0084] Module M1: Classify the backup tables in the database based on business processing logic;

[0085] Specifically, in the module M1:

[0086] Classify the backup tables in the database based on business processing logic according to data management, and the classifications of the tables obtained include: F1 algorithm, F2 algorithm, I algorithm, and F5 algorithm;

[0087] Among them, the F1 algorithm is full - volume insertion. When each job runs, all data in the current data table is deleted, and then the operation result of the current business date is retained, without audit fields;

[0088] The F2 algorithm is incremental update. Update the records in the data table according to the primary key. This type of data processing job has the characteristics of storing historical data and retaining its latest status except for the primary key, and has audit fields;

[0089] The I algorithm is direct insertion. Append the records of the selected fields to the end of the data table, and has audit fields;

[0090] The F5 algorithm is historical zipper. When there is an update to the data except for the primary key, update the end_date of the original record to the day before the current business date; then add a new record with the start_date set to the current business date, record the status values of the corresponding fields, and set the end_date field.

[0091] Module M2: Sub - divide the classified tables into different ordered time granularities through time granularity;

[0092] Specifically, in the module M2:

[0093] Sub - divide the classification of the tables into different ordered time granularities according to the actual situation through time granularity, which are: daily, weekly, and monthly;

[0094] Among them, in the case of table types F1, F2, I, and F5, perform daily frequency; in the case of table types I and F5, perform weekly frequency backup; in the case of table types F1, F2, I, and F5, perform monthly frequency backup.

[0095] Module M3: Determine the backup methods as incremental backup, full - volume backup, and overall backup according to the classification of the tables and time granularity;

[0096] Specifically, in the module M3:

[0097] Determine the backup methods as incremental backup, full - volume backup, and overall backup according to the determined table classification and time granularity;

[0098] The amount of data processed by the F1 and F2 algorithms is relatively small, so incremental backup is performed daily. The amount of data processed by the I and F5 algorithms is relatively large, so full backup is performed once a week on the basis of daily incremental backup. All data in the database is backed up as a whole once a month.

[0099] For F1 and F2, the backup frequency is daily and the backup method is incremental backup. Incremental backup means that after a full backup or the last incremental backup, each subsequent backup only needs to back up the data that has been added and modified compared with the previous one, and the data can be restored through the most recent full backup and all incremental backups.

[0100] For tables classified as I and F5, the backup frequency is weekly, and the backup method is determined to be full backup. Full backup is a complete copy of all data at a certain point in time, and only the last full backup is needed for recovery;

[0101] The tables are classified as F1, F2, I and F5, the backup frequency is monthly, and the backup method is determined to be overall backup. Overall backup is to perform a full backup of all data in the database at a certain point in time.

[0102] Module M4: performs distributed backup and uses parallel writing to exclusively use the single write bandwidth.

[0103] Specifically, in the module M4:

[0104] Distributed backup is performed through parallel writing, with a single write bandwidth exclusive. The number of MPP cluster nodes and backup nodes is in a preset ratio. The backup node file directory is mounted on the cluster node through the NFS service. As the number of MPP nodes increases, the number of backup servers is increased according to the preset ratio.

[0105] Embodiment 3:

[0106] Embodiment 3 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0107] The content of this invention is a backup deployment solution based on an open MPP cluster (GREENPLUM, GAUSSDB, etc.) of a data warehouse. It makes full use of the distributed characteristics of the MPP cluster. The backup deployment solution also adopts a distributed form. Without the need for professional backup equipment, the distributed deployment backup of ordinary PC physical machines can complete the daily full backup in hours (the specific time needs to be calculated according to different cluster bandwidths, single-machine capacity, backup node ratio, etc.). At the same time, the backup nodes can be expanded linearly according to the MPP data nodes, thereby ensuring the high efficiency of the MPP cluster's entire database backup.

[0108] A distributed backup solution for an open MPP database based on a data warehouse solves the backup efficiency problem by adopting a distributed backup architecture and supports horizontal expansion of backup nodes. The specific solution includes:

[0109] Step 1:

[0110] Based on data management, the tables to be backed up in the database can be classified according to business processing logic, resulting in four classifications of tables: F1 algorithm, F2 algorithm, I algorithm, and F5 algorithm. The F1 algorithm is full insert, which deletes all data in the current data table during each job run and then retains the operation result of the current business date without audit fields, and is suitable for the production of daily reports; the F2 algorithm is incremental update, which updates the records in the data table according to the primary key. This type of data processing job has the characteristics of storing historical data and retaining its latest state except for the primary key, and has audit fields, and is suitable for data tables such as merchant information tables; the I algorithm is direct insert, which appends the records of the selected fields to the end of the data table and has audit fields, and is mainly used for data integration of transaction flow tables; the F5 algorithm is historical zipper. When there is an update to the data except for the primary key, the end_date of the original record is updated to the day before the current business date. Subsequently, a new record is added with the start_date set to the current business date, and the status value of the corresponding field is recorded, and the end_date field is set to 30001231 (open chain). This algorithm saves historical data and has audit fields, and is suitable for data tables such as customer time deposit information tables.

[0111] Step 2:

[0112] The classification of the tables in Step 1 is further divided into three different ordered time granularities according to the actual situation by time granularity, namely: daily, weekly, and monthly. Among them, when the table type is F1, F2, I, F5, a daily frequency is required; when the table type is I, F5, a weekly frequency backup is required; when the table type is F1, F2, I, F5, a monthly frequency backup is required.

[0113] Step 3:

[0114] Based on the table classification and time granularity determined in Step 1 and Step 2, the backup method is determined to be incremental backup, full backup, and overall backup. For the data processed by algorithms F1 and F2 with a relatively small data volume, incremental backup is performed daily; for the data processed by algorithms I and F5 with a relatively large data volume, in addition to incremental backup daily, a full backup is performed once a week; all the data in the database is overall backed up once a month. For the table classification of F1 and F2, the backup frequency is daily, and the backup method is determined to be incremental backup. Incremental backup is performed after a full backup or the previous incremental backup. For each subsequent backup, only the data that has increased or been modified compared to the previous one needs to be backed up. Restoration can be achieved through the most recent full backup and all incremental backups. It has the advantages of fast backup speed and less storage space occupied, and is suitable for data backup in scenarios with high frequency, multiple batches, and small data volumes. For the table classification of I and F5, the backup frequency is weekly, and the backup method is determined to be full backup. Full backup is a complete copy of all data at a certain point in time, and restoration can be achieved only with the previous full backup. It has the advantage of fast restoration speed and is suitable for data backup in scenarios with low frequency, small batches, and large data volumes. For the table classification of F1, F2, I, and F5, the backup frequency is monthly, and the backup method is determined to be overall backup. Overall backup is a full backup of all the data in the database at a certain point in time, which is suitable for disaster recovery and security backup of the entire database.

[0115] Step 4:

[0116] After all the previous steps are completed, distributed backup is performed. Through the parallel writing method, each write bandwidth is exclusive and does not conflict with others. Since flexible backup methods are adopted for different data at different time granularities, not only the backup efficiency is improved, but also the hardware requirements for backup are greatly reduced. The MPP cluster configures backup nodes according to the ratio of the number of nodes / backup nodes = 5:1. The physical machines of the backup nodes can purchase inexpensive servers with large storage space, small CPU, and small memory according to requirements, and the network bandwidth ensures full-duplex 10 Gigabit. Mount the backup node file directory on the cluster nodes through the NFS service, so that 5 MPP nodes can write files in parallel, thereby realizing the distributed backup mechanism and supporting parallel backup. Subsequently, as the number of MPP nodes increases, the number of backup servers is increased in the same ratio of 5:1, which can ensure that the backup time will not increase linearly with the increase of the data volume.

[0117] 6. Backup test results:

[0118] Based on this backup architecture, the data warehouse project team of Industrial Bank of China carried out a production environment deployment. Currently, there are 23 MPP data nodes, 2 MPP management nodes, and 5 backup nodes in the production MPP cluster. The specific test results are as follows:

[0119] Backup Date Elapsed Time Backup File Size Concurrency 20210928 6 hours 18 minutes 59 seconds 16.35TB 1 20210929 1 hour 21 minutes 22 seconds 16.44TB 10 20211011 1 hour 14 minutes 20 seconds 17.34TB 20 20211011 6 minutes 20 seconds 1.44TB 10 20211014 9 minutes 30 seconds 1.97TB 10

[0120] Judging from the test results, there is little difference between 10 concurrent backups and 20 backups. Based on the reference data (20211014), estimate the transmission speed of a single file server. According to the formula:

[0121] Backup Generated File Size / Backup Elapsed Time / Number of File Servers = Transfer Speed per File Server

[0122] Calculate the transmission speed of a single file server as:

[0123] (1.97TB * 1024 * 1024) / 570S / 5 times compression ratio = 724.8MB / S

[0124] Based on the reference data, estimate the time-consuming of cluster backup. For a single unit, according to the formula:

[0125]

[0126] Note:

[0127] 1. Since the backup files are compressed files, the actual size of the data files backed up by the cluster needs to be divided by the compression ratio. After testing, the compression ratio is about 5 times.

[0128] 2. Calculate the capacity of 1 MPP data node according to 24 pieces of 2.4TB capacity. The database space is about 45TB. Considering that MPP has two copies of data plus 70% high-availability space, the available capacity of 5 MPP data nodes is (45TB / 2) * 70% * 5 = 78.75TB

[0129] Calculate the time-consuming of full database backup as:

[0130] (78.75 * 1024 * 1024MB / 5) / (724.8MB / S) / 3600S / H = 6.329H

[0131] The estimated time-consuming of full database backup is 6.329H

[0132] Therefore, without considering the data compression of the database itself, as long as distributed backup is carried out according to the 5:1 architecture and there is a dual 10Gigabit Ethernet connection between the MPP cluster and the backup nodes, the overall backup time will not exceed 6.329H.

[0133] Those skilled in the art know that in addition to implementing the systems, devices and their respective modules provided by the present invention in the form of pure computer-readable program codes, the method steps can be logically programmed to enable the systems, devices and their respective modules provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same program. Therefore, the systems, devices and their respective modules provided by the present invention can be considered as a kind of hardware components, and the modules included therein for implementing various programs can also be regarded as the structures within the hardware components; the modules for implementing various functions can also be regarded as either software programs for implementing methods or structures within hardware components.

[0134] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A distributed backup method for an open MPP database based on a data warehouse, characterized in that, Including: Step S1: Classify the backup tables in the database based on the business processing logic; Step S2: Subdivide the classified tables into different ordered time granularities according to the time granularity; Step S3: Determine the backup methods as incremental backup, full backup, and overall backup according to the classification of the tables and the time granularity; Step S4: Perform distributed backup, and each write bandwidth is exclusive through parallel writing; In the said Step S1: Classify the backup tables in the database based on the business processing logic according to data management, and the obtained table classifications include: F1 algorithm, F2 algorithm, I algorithm, and F5 algorithm; Among them, the F1 algorithm is full insertion. When each job runs, all data in the current data table is deleted, and then the operation result of the current business date is retained, without an audit field; The F2 algorithm is incremental update. It updates the records in the data table according to the primary key, has the characteristics of storing historical data and retaining its latest state except the primary key, and has an audit field; The I algorithm is direct insertion. Append the records of the selected fields to the end of the data table, and have an audit field; The F5 algorithm is historical zipper. When there is an update to the data except the primary key, update the end_date of the original record to the day before the current business date; then add a new record, set the start_date to the current business date, record the status values of the corresponding fields, and set the end_date field; In the said Step S3: Determine the backup methods as incremental backup, full backup, and overall backup according to the determined table classification and time granularity; The data volume of the data processed by the F1 and F2 algorithms is small, and incremental backup is performed daily; the data volume of the data processed by the I and F5 algorithms is large. On the basis of performing incremental backup daily, a full backup is performed once a week; all data in the database is overall backed up once a month; For the classification of F1 and F2, the backup frequency is daily, and the backup method is determined as incremental backup; incremental backup is after a full backup or the previous incremental backup. For each subsequent backup, only the data that has increased and been modified compared with the previous one needs to be backed up, and it is restored through the most recent full backup and all incremental backups; For the classification of I and F5 of the tables, the backup frequency is weekly, and the backup method is determined as full backup. Full backup is a complete copy of all data at a certain point in time, and only the previous full backup is required for restoration; For the classification of F1, F2, I, and F5 of the tables, the backup frequency is monthly, and the backup method is determined as overall backup. Overall backup is to perform a full backup of all data in the database at a certain point in time.

2. The distributed backup method for an open MPP database based on a data warehouse according to claim 1, characterized in that In the said Step S4: Perform distributed backup, and each write bandwidth is exclusive through parallel writing; the number of MPP cluster nodes and backup nodes is a preset ratio; mount the backup node file directory on the cluster nodes through the NFS service; Increase the number of backup servers according to a preset ratio as the number of MPP nodes increases.

3. A distributed backup system based on an open MPP database of a data warehouse, characterized in that, Including: Module M1: Classify the backup tables in the database based on the business processing logic; Module M2: Subdivide the classified table into different ordered time granularities according to the time granularity. Module M3: Determine the backup method as incremental backup, full backup, and overall backup according to the classification of the table and the time granularity. Module M4: Perform distributed backup, and each write bandwidth is exclusive through the parallel write method. In the said Module M1: Classify the backup tables in the database based on the business processing logic according to data management, and the obtained table classifications include: F1 algorithm, F2 algorithm, I algorithm, and F5 algorithm. Among them, the F1 algorithm is full insertion. When each job runs, all data in the current data table is deleted, and then the operation result of the current business date is retained without audit fields. The F2 algorithm is incremental update. Update the records in the data table according to the primary key, with the characteristics of storing historical data and retaining its latest state except the primary key, and with audit fields. The I algorithm is direct insertion. Append the records of the selected fields to the end of the data table, and with audit fields. The F5 algorithm is historical zipper. When there is an update to the data except the primary key, update the end_date of the original record to the day before the current business date; then add a new record, set the start_date to the current business date, record the status value of the corresponding field, and set the end_date field. In the said Module M3: Determine the backup method as incremental backup, full backup, and overall backup according to the determined table classification and time granularity. The data volume of the data processed by the F1 and F2 algorithms is small, and incremental backup is performed daily; the data volume of the data processed by the I and F5 algorithms is large. On the basis of performing incremental backup daily, a full backup is performed once a week; all data in the database is subject to an overall backup once a month. For the classification of F1 and F2, the backup frequency is daily, and the backup method is determined as incremental backup. Incremental backup is after a full backup or the previous incremental backup. For each subsequent backup, only the data that has increased and been modified compared with the previous one needs to be backed up, and it is restored through the most recent full backup and all incremental backups. For the classification of I and F5 of the table, the backup frequency is weekly, and the backup method is determined as full backup. Full backup is to make a complete copy of all data at a certain point in time, and only the previous full backup is required for restoration. For the classification of F1, F2, I, and F5 of the table, the backup frequency is monthly, and the backup method is determined as overall backup. Overall backup is to perform a full backup of all data in the database at a certain point in time.

4. The distributed backup system based on the data warehouse open MPP database according to claim 3, characterized in that, In the said Module M4: Perform distributed backup, and each write bandwidth is exclusive through the parallel write method; the number of MPP cluster nodes and backup nodes is in a preset ratio; mount the backup node file directory on the cluster node through the NFS service. Increase the number of backup servers in accordance with the preset ratio as the number of MPP nodes increases.

Images