A data backup and disaster recovery method and system based on data lake platform

Abstract

The present invention relates to the field of data processing technology, and in particular to a data backup and disaster recovery system based on a data lake platform, comprising the following steps: a data backup module for backing up data in a data lake; a snapshot module for generating corresponding snapshots when data in the data lake is changed; a disaster recovery module for restoring the backed-up data and the snapshots; a control module for determining whether to issue an information warning according to whether anomalies occur in the data backup process and the disaster recovery process, and for adjusting the data backup method according to the average duration of backup data recovery or performing a determination function for the validity of a tag for metadata information in combination with the data difference byte ratio of metadata information, and determining a tag problem response according to the determination result of the tag validity. The present invention improves the validity of data backup.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data backup and disaster recovery method and system based on a data lake platform. Background Art

[0002] The data lake platform can help users easily build and manage large-scale data lakes, and provide efficient data incremental modification, fast data access and analysis, and complete data lake management and metadata control functions.

[0003] Current data backup and disaster recovery technologies for data lake platforms usually use a periodic full backup strategy, which means that all data in the entire data lake is backed up at regular intervals and stored in different locations; however, this approach often leads to outdated and inefficient backup data when rapid incremental updates are made in the data lake. In addition, this approach may require a large amount of storage space and bandwidth, which increases costs and complexity.

[0004] Chinese patent publication number CN106407045A discloses a data disaster recovery method, including: backing up metadata on a work site of a server virtualization system; real-time monitoring of the work site; when a work site is monitored to fail, using the metadata backup file, performing corresponding data recovery processing on the standby site, and then starting the standby site. In this application, it is necessary to back up metadata on the work site. When a work site is monitored to fail, the metadata backup file will be used to perform data recovery processing on the standby site, and then the standby site that has undergone data recovery processing will be used to replace the work process of the work site that failed before, thereby reducing the service interruption time after the system fails, and reducing the adverse effects of the system failure of the server virtualization system on users. In addition, the application also discloses a data disaster recovery device and a server virtualization system including the device. It can be seen that the data disaster recovery method has the problem that due to the huge amount of data in the data lake, it takes a long time to back up the data. If data loss occurs when backing up the data in sequence, the timeliness of important data backup will be reduced or the backup data at the time of data loss will be garbled, resulting in an increase in the time of data recovery, thereby causing the effectiveness of data backup to decrease. Summary of the invention

[0005] To this end, the present invention provides a data backup and disaster recovery system based on a data lake platform, which is used to overcome the problem in the prior art that due to the huge amount of data in the data lake, it takes a long time to back up the data. If data loss occurs when backing up the data in sequence, the timeliness of important data backup will be reduced or the backup data at the time of data loss will be garbled, resulting in an increase in the time for data recovery, thereby causing a decrease in the effectiveness of data backup.

[0006] To achieve the above-mentioned purpose, the present invention provides a data backup and disaster recovery system based on a data lake platform, comprising: a data backup module, used to back up all data in the data lake platform; a snapshot module, which is connected to the data backup module, used to generate corresponding snapshots when data in the data lake changes, including a version creation component used to create several versions of the data lake and a marking component connected to the version creation component for marking metadata information in each version; a disaster recovery module, which is respectively connected to the data backup module and the snapshot module, used to recover the backup data and the snapshot; a control module, which is respectively connected to the data backup module, the snapshot module and the disaster recovery module, used to determine whether to issue an information warning according to whether an abnormality occurs in the data backup process and the disaster recovery process, and, when it is determined that the validity of the data backup does not meet the requirements according to the average time of the backup data recovery, adjust the data backup mode or perform a determination function for the marking validity of the metadata information in combination with the data difference byte ratio of the metadata information, and determine a marking problem response according to the determination result of the marking validity, wherein the marking problem response includes adjusting the warning trigger threshold or adjusting the number of marking types of the metadata information according to the loss ratio of the backup data.

[0007] Furthermore, the data backup module includes:

[0008] A capture component is used to capture the changed data in all data in the data lake platform;

[0009] The backup storage component is connected to the capture component and is used to store the captured data.

[0010] Furthermore, the control module obtains the recovery time of several backup data of the same data volume and the total number of backup data recovery times, and calculates the average recovery time of the backup data according to the obtained results.

[0011] The control module determines that the validity of the data backup does not meet the requirements under the preset first average duration condition or the preset second average duration condition;

[0012] The control module adjusts the sequential backup mode to the priority backup mode when only the preset first time period is met;

[0013] Among them, the preset first average duration condition is that the average duration of the backup data recovery is greater than the preset second average duration; the preset second average duration condition is that the average duration of the backup data recovery is greater than the preset first average duration and less than or equal to the preset second average duration.

[0014] Furthermore, the sequential backup is to back up all the data in the data lake platform in order of data volume, and the priority backup is to back up all the data in the data lake platform according to the priority of the data.

[0015] Furthermore, the calculation formula for the average duration of the backup data recovery is:

[0016]

[0017] Among them, G is the average time for restoring backup data, H x is the duration of restoring the x-th backup data, and y is the total number of times the backup data is restored, wherein the amount of data in each backup is the same.

[0018] Furthermore, the control module preliminarily determines that the marking validity of the metadata information does not meet the requirements under the preset second average duration condition, and performs a determination function for the marking validity of the metadata information in combination with the data difference byte ratio of the metadata information, including:

[0019] The data bytes that are inconsistent between the marked metadata information and the data in the data lake platform and the total bytes of the marked metadata information are obtained respectively, and the proportion of the data difference bytes of the metadata information is calculated based on the obtained results, where:

[0020] The control module secondarily determines that the marking validity of the metadata information does not meet the requirement under a preset first difference byte ratio condition or a preset second difference byte ratio condition, and executes the marking problem response;

[0021] The control module reduces the warning trigger threshold when only the preset first difference byte ratio is met.

[0022] Among them, the preset first difference byte ratio condition is that the data difference byte ratio of the metadata information is greater than the preset second difference byte ratio; the preset second difference byte ratio condition is that the data difference byte ratio of the metadata information is greater than the preset first difference byte ratio and is less than or equal to the preset second difference byte ratio.

[0023] Furthermore, the reduction range of the warning triggering threshold is determined by the difference between the data difference byte ratio of the metadata information and the preset second difference byte ratio.

[0024] Furthermore, the control module preliminarily determines that the comprehensiveness of the metadata information mark does not meet the requirement under the preset second difference byte amount ratio condition, and obtains the lost bytes of the backup data and the total bytes of the backup data respectively, and calculates the lost amount ratio of the backup data according to the obtained results.

[0025] The control module determines for the second time that the comprehensiveness of the metadata information mark does not meet the requirement under the preset loss amount ratio condition, and increases the number of the mark types of the metadata information;

[0026] The preset loss amount ratio condition is that the loss amount ratio of the backup data is greater than the preset loss amount ratio.

[0027] Furthermore, the increase range of the number of the marking types of the metadata information is determined by the difference between the missing amount ratio of the backup data and the preset loss amount ratio.

[0028] The present invention also provides a data backup and disaster recovery method based on a data lake platform, comprising the following steps:

[0029] Regularly back up the data in the data lake;

[0030] Creating several versions of the data lake and marking metadata information in each version;

[0031] When a disaster occurs, the backed-up data and marked metadata information are restored, and the data backup process and disaster recovery process are monitored in real time;

[0032] Obtain the recovery time of several backup data with the same data volume and the total number of backup data recovery times;

[0033] Determining whether the validity of the data backup meets the requirements based on the average recovery time of the backup data;

[0034] If it is determined that the validity does not meet the requirements, the data backup method is adjusted or the determination function of the marking validity of the metadata information is performed in combination with the data difference byte ratio of the metadata information;

[0035] Determining a response to a marking question based on the determination result of the marking validity;

[0036] The marking problem response includes adjusting an early warning trigger threshold or adjusting the number of marking types of metadata information according to the proportion of backup data loss.

[0037] Compared with the prior art, the beneficial effect of the present invention lies in that the system of the present invention adjusts the data to be backed up in sequence according to the average time of backup data recovery by setting a data backup module, a disaster recovery module, a snapshot module, a mirror module and a control module. Due to the huge amount of data in the data lake, it takes a long time to back up the data. If data loss occurs when backing up the data in sequence, the timeliness of important data backup will be reduced or the backup data at the time of data loss will be garbled, resulting in an increase in the time of data recovery. The data backup is adjusted to be backed up according to the priority of the data to ensure that important data is not lost, thereby improving the effectiveness of data backup; according to the proportion of data difference bytes of metadata information, the warning trigger threshold is reduced, the data source in the data lake changes, the original metadata information cannot fully cover the new data structure in time, the metadata tag is inaccurate, the tagged metadata information is inconsistent with the data in the data lake, the tag effectiveness of the metadata information is reduced, and the warning trigger threshold is reduced to ensure that a warning is issued in time when the data is inconsistent, and the metadata information is updated in time, thereby improving the tag effectiveness of the metadata information, and achieving the improvement of the effectiveness of data backup.

[0038] Furthermore, the system of the present invention sets a preset first average duration and a preset second average duration, and adjusts the data from being backed up in sequence to being backed up according to the average duration of restoring the backup data to being backed up according to the priority of the data. Since the amount of data in the data lake is huge, it takes a long time to back up the data. If data is lost when the data is backed up in sequence, the timeliness of the important data backup may be reduced or the backup data at the time of data loss may be garbled, resulting in an increase in the duration of data recovery. The data backup is adjusted to be backed up according to the priority of the data to ensure that important data is not lost, thereby improving the effectiveness of the data backup, and further achieving an improvement in the effectiveness of the data backup.

[0039] Furthermore, the system of the present invention reduces the warning trigger threshold according to the data difference byte ratio of metadata information by setting a preset first difference byte ratio and a preset second difference byte ratio. When the data source in the data lake changes, the original metadata information cannot fully cover the new data structure in time, the metadata marking is inaccurate, the marked metadata information is inconsistent with the data in the data lake, and the marking validity of the metadata information decreases. The warning trigger threshold is reduced to ensure that a warning is issued in time when data inconsistencies occur, and the metadata information is updated in time, thereby improving the marking validity of the metadata information, and further improving the effectiveness of data backup.

[0040] Furthermore, the system of the present invention increases the number of metadata information tag types according to the backup data loss ratio by setting a preset loss ratio. Due to the long-term resource usage limitation of the system, there are omissions in the types of metadata information tagged, lacking types such as data lifecycle tags and metadata keyword tags, resulting in the inability to accurately manage and track data during archiving and deletion, thereby causing data loss and insufficient comprehensiveness of metadata information tags. The number of metadata information types is increased to reduce omissions in the types of metadata information tagged, thereby improving the comprehensiveness of metadata information tags, and further achieving improved effectiveness of data backup. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is an overall structural block diagram of a data backup and disaster recovery system based on a data lake platform according to an embodiment of the present invention;

[0042] Figure 2 This is a specific structural block diagram of a data backup module of a data backup and disaster recovery system based on a data lake platform according to an embodiment of the present invention;

[0043] Figure 3 This is a specific structural block diagram of a snapshot module of a data backup and disaster recovery system based on a data lake platform according to an embodiment of the present invention;

[0044] Figure 4 The figure is an overall flow chart of the data backup and disaster recovery method based on the data lake platform according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0047] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0048] It should be pointed out that the data in this embodiment are obtained by the data backup and disaster recovery system based on the data lake platform of the present invention before the current use according to the historical data and the data statistics in the corresponding test process, the test experiment and the comprehensive analysis based on the experimental results; the data backup and disaster recovery system based on the data lake platform of the present invention is based on the average duration of the backup data recovery of 1463 cases that are cumulatively counted, detected and calculated within 86 days, the proportion of the data difference bytes of the metadata information and the proportion of the loss of the backup data, and comprehensively determines the values ​​of the various preset parameter standards for the data backup and disaster recovery system based on the data lake platform. It can be understood by those skilled in the art that the data backup and disaster recovery system based on the data lake platform of the present invention can determine the single parameter by selecting the value with the highest proportion as the preset standard parameter according to the data distribution, as long as the system of the present invention can clearly define the different specific situations in the single determination process through the obtained values.

[0049] See also Figure 1 , Figure 2 , Figure 3 as well as Figure 4 As shown, they are respectively an overall structural block diagram of a data backup and disaster recovery system based on a data lake platform, a specific structural block diagram of a data backup module, a specific structural block diagram of a snapshot module, and an overall flow chart of a data backup and disaster recovery method based on a data lake platform according to an embodiment of the present invention. A data backup and disaster recovery system based on a data lake platform according to the present invention comprises:

[0050] Data backup module, used to back up all data in the data lake platform;

[0051] A snapshot module, which is connected to the data backup module and is used to generate a corresponding snapshot when the data in the data lake changes, including a version creation component for creating several versions of the data lake and a marking component connected to the version creation component for marking metadata information in each version;

[0052] A disaster recovery module, which is connected to the data backup module and the snapshot module respectively, and is used to restore the backup data and the snapshot;

[0053] A control module is connected to the data backup module, the snapshot module and the disaster recovery module respectively, and is used to determine whether to issue an information warning according to whether an abnormality occurs in the data backup process and the disaster recovery process, and, when it is determined that the validity of the data backup does not meet the requirements according to the average time of the backup data recovery, adjust the data backup method or perform a determination function on the validity of the mark of the metadata information in combination with the data difference byte ratio of the metadata information, and determine a mark problem response according to the determination result of the validity of the mark,

[0054] The marking problem response includes adjusting an early warning trigger threshold or adjusting the number of marking types of metadata information according to the proportion of backup data loss.

[0055] Specifically, the data backup module includes:

[0056] A capture component is used to capture the changed data in all data in the data lake platform;

[0057] The backup storage component is connected to the capture component and is used to store the captured data.

[0058] Specifically, the capture component uses change data capture technology to monitor data changes in the data lake platform in real time. When data changes are detected, the change data capture technology captures these changes and records them.

[0059] Specifically, the version creation component creates several versions of the data lake as copies of the data in the data lake platform at a set time point.

[0060] Specifically, each replica is an independent data set.

[0061] Specifically, metadata information includes the creation time of the snapshot, the amount of data in the snapshot, and the source of the data in the data lake.

[0062] Specifically, the data sources in the data lake include the cloud, big data platforms, and social platforms.

[0063] Specifically, the data backup and disaster recovery system based on the data lake platform of the present invention also includes a mirror module connected to the disaster recovery module and the monitoring module respectively for copying the data in the data lake to a remote target location.

[0064] Specifically, the mirroring module uses a streaming replication method to replicate data in the data lake to a remote target location.

[0065] It can be understood that streaming replication is a continuous data transfer method that can copy data from a data lake location to a remote target location in real time or near real time.

[0066] Specifically, the remote target location is another storage location different from the original storage location of the data lake, including a local data center, a dedicated disaster recovery center, and a mobile hard disk.

[0067] Specifically, several versions of the data lake are the states of the data in the data lake platform at different points in time.

[0068] Specifically, the state characteristic data of the data at different time points include the change time of the data, the size of the data, and the number of times the data is called.

[0069] Specifically, anomalies in the data backup process include data loss during data backup, data garbled during data backup, and data duplicate backup during data backup.

[0070] Specifically, abnormalities in the disaster recovery process include data recovery process being stuck for up to 10 seconds, data corruption, and backup data loss.

[0071] In implementation, the system of the present invention adjusts the data backup in sequence to the data priority according to the average time of backup data recovery by setting a data backup module, a disaster recovery module, a snapshot module, a mirror module and a control module. Due to the huge amount of data in the data lake, it takes a long time to back up the data. If data loss occurs when backing up the data in sequence, the timeliness of important data backup will be reduced or the backup data at the time of data loss will be garbled, resulting in an increase in the time of data recovery. The data backup is adjusted to be backed up according to the data priority to ensure that important data is not lost, thereby improving the effectiveness of data backup; according to the proportion of data difference bytes of metadata information, the warning trigger threshold is reduced. The data source in the data lake changes, the original metadata information cannot fully cover the new data structure in time, the metadata tag is inaccurate, the tagged metadata information is inconsistent with the data in the data lake, and the tag effectiveness of the metadata information is reduced. The warning trigger threshold is reduced to ensure that a warning is issued in time when the data is inconsistent, and the metadata information is updated in time, thereby improving the tag effectiveness of the metadata information, thereby improving the effectiveness of data backup.

[0072] Specifically, the control module obtains the recovery time of several backup data of the same data volume and the total number of backup data recovery times, and calculates the average recovery time of the backup data according to the obtained results.

[0073] The control module determines that the validity of the data backup does not meet the requirements under the preset first average duration condition or the preset second average duration condition;

[0074] The control module adjusts the sequential backup mode to the priority backup mode when only the preset first time period is met;

[0075] Among them, the preset first average duration condition is that the average duration of the backup data recovery is greater than the preset second average duration; the preset second average duration condition is that the average duration of the backup data recovery is greater than the preset first average duration and less than or equal to the preset second average duration.

[0076] Specifically, the sequential backup is to back up all the data in the data lake platform in order of data volume, and the priority backup is to back up all the data in the data lake platform according to the priority of the data.

[0077] Specifically, the calculation formula for the average duration of restoring the backup data is:

[0078]

[0079] Among them, G is the average time for restoring backup data, H x is the duration of restoring the x-th backup data, and y is the total number of times the backup data is restored, wherein the amount of data in each backup is the same.

[0080] Preferably, the priorities of the data include a first level, a second level and a third level.

[0081] Preferably, when the amount of data that has changed since the last backup is greater than or equal to 20KB, the priority of the data is the first level; when the amount of data that has changed since the last backup is greater than 10KB and less than 20KB, the priority of the data is determined to be the second level; when the amount of data that has changed since the last backup is less than or equal to 10KB, the priority of the data is the third level.

[0082] Specifically, when backing up data according to their priority, the backup order is first level, second level, and third level.

[0083] In implementation, the system of the present invention sets a preset first average duration and a preset second average duration, and adjusts the data from being backed up in sequence to being backed up according to the average duration of backup data recovery. Due to the huge amount of data in the data lake, it takes a long time to back up the data. If data loss occurs when the data is backed up in sequence, the timeliness of the important data backup may be reduced or the backup data at the time of data loss may be garbled, resulting in an increase in the data recovery time. The data backup is adjusted to be backed up according to the data priority to ensure that important data is not lost, thereby improving the effectiveness of the data backup, and further achieving the improvement of the effectiveness of the data backup.

[0084] Specifically, the control module preliminarily determines that the marking validity of the metadata information does not meet the requirements under the preset second average duration condition, and performs a determination function for the marking validity of the metadata information in combination with the data difference byte ratio of the metadata information, including:

[0085] The data bytes that are inconsistent between the marked metadata information and the data in the data lake platform and the total bytes of the marked metadata information are obtained respectively, and the proportion of the data difference bytes of the metadata information is calculated based on the obtained results, where:

[0086] The control module secondarily determines that the marking validity of the metadata information does not meet the requirement under a preset first difference byte ratio condition or a preset second difference byte ratio condition, and executes the marking problem response;

[0087] The control module reduces the warning trigger threshold when only the preset first difference byte ratio is met.

[0088] Among them, the preset first difference byte ratio condition is that the data difference byte ratio of the metadata information is greater than the preset second difference byte ratio; the preset second difference byte ratio condition is that the data difference byte ratio of the metadata information is greater than the preset first difference byte ratio and is less than or equal to the preset second difference byte ratio.

[0089] Specifically, the reduction range of the warning triggering threshold is determined by the difference between the data difference byte ratio of the metadata information and the preset second difference byte ratio.

[0090] Specifically, the calculation formula for the percentage of data difference bytes in metadata information is:

[0091]

[0092] Among them, M is the percentage of data difference bytes of metadata information, D1 is the number of data bytes that are inconsistent between the marked metadata information and the data in the data lake platform, and D2 is the total number of data bytes of the marked metadata information.

[0093] In implementation, the preset selection range of the first difference byte ratio M1 is 0.1-0.25, and the preset selection range of the second difference byte ratio M2 is 0.3-0.5. When the preset selection value of the second difference byte ratio is greater than 0.3, the corresponding influencing factors may change. For example, the data difference byte ratio of the metadata information is too large, and the difference between the data in the snapshot and the data in the data lake is too large, resulting in the data difference byte ratio of the metadata information having too much influence on the marking validity of the metadata information.

[0094] Specifically, the warning trigger thresholds include the number of bytes that are inconsistent between the metadata information and the data on the data lake platform, the amount of data that is inconsistent between the backed-up data and the data on the data lake platform, and the number of times the same data is repeatedly restored during the recovery process of the backed-up data.

[0095] Specifically, issuing an information warning includes simultaneously issuing a text message warning, a sound warning, and a color warning.

[0096] Specifically, the sound warning is to emit a warning sound to prompt that abnormalities occur in the data backup process and disaster recovery process.

[0097] In this embodiment, the warning triggering threshold is the total number of bytes that are inconsistent between the metadata information and the data on the data lake platform.

[0098] Specifically, the calculation formula for the reduced warning trigger threshold is:

[0099]

[0100] Among them, T' is the reduced warning trigger threshold, T is the current warning trigger threshold, Z is the minimum unit difference byte ratio for adjusting the warning trigger threshold, and t is the warning trigger threshold when the data difference byte ratio of the metadata information is the minimum unit difference byte ratio.

[0101] In practice, the value of Z is generally 0.2 to 0.3, and preferably, the preferred embodiment of Z is 0.3.

[0102] In practice, the value of t is generally set at 1KB to 3KB. Preferably, considering that too many warning trigger thresholds may result in failure to issue a warning in time, leading to data loss, the preferred embodiment of t is 2.

[0103] For example, when the current warning trigger threshold is T=5KB and the percentage of the data difference bytes of the metadata information is M=0.6, the reduced warning trigger threshold is:

[0104]

[0105] In implementation, the system of the present invention sets a preset first difference byte ratio and a preset second difference byte ratio, and reduces the warning trigger threshold according to the data difference byte ratio of the metadata information. The data source in the data lake changes, the original metadata information cannot fully cover the new data structure in time, the metadata tag is inaccurate, the tagged metadata information is inconsistent with the data in the data lake, and the tag validity of the metadata information decreases. The warning trigger threshold is reduced to ensure that a warning is issued in time when data inconsistencies occur, and the metadata information is updated in time, thereby improving the tag validity of the metadata information, and further improving the effectiveness of data backup.

[0106] Specifically, the control module preliminarily determines that the comprehensiveness of the metadata information mark does not meet the requirement under the preset second difference byte amount ratio condition, and obtains the lost byte amount of the backup data and the total byte amount of the backup data respectively, and calculates the lost amount ratio of the backup data according to the obtained results.

[0107] The control module determines for the second time that the comprehensiveness of the metadata information mark does not meet the requirement under the preset loss amount ratio condition, and increases the number of the mark types of the metadata information;

[0108] The preset loss amount ratio condition is that the loss amount ratio of the backup data is greater than the preset loss amount ratio.

[0109] Specifically, the increase in the number of the marking types of the metadata information is determined by the difference between the missing amount ratio of the backup data and the preset loss amount ratio.

[0110] Specifically, the calculation formula for the percentage of backup data loss is:

[0111]

[0112] Among them, R is the percentage of backup data loss, B1 is the number of lost bytes of backup data, and B2 is the total number of bytes of backup data.

[0113] It is understandable that, in practice, the selection range of the preset loss ratio R0 is 0.2 to 0.4. When the selected value of the preset loss ratio is less than 0.3, the corresponding influencing factors may change. For example, when the loss ratio of backup data is too high, the difference between the backup data and the data in the data lake is too large, resulting in too much impact of the data loss ratio on the comprehensiveness of the metadata information tag.

[0114] Specifically, the calculation formula for the number of tag types of the increased metadata information is:

[0115]

[0116] Among them, K' is the number of tag types of metadata information after the increase, K is the number of tag types of current metadata information, c is the minimum unit loss ratio for adjusting the number of tag types of metadata information, and k is the number of tag types of metadata information when the loss ratio of backup data is the minimum unit loss ratio.

[0117] Specifically, the marking types of metadata information include data recovery time, data change record, and data change time.

[0118] Optionally, the value of c is usually 0.3 to 0.4, and preferably, the preferred embodiment of c is 0.3.

[0119] Optionally, the optional range of k is 1 to 3. Preferably, considering that when the number of tag types of metadata information adjusted at a single time is greater than 2, the effectiveness of the tags may decrease, the preferred embodiment of k is 2.

[0120] For example, when the number of tag types of the current metadata information is K=3 and the proportion of backup data loss is R=0.6, the number of tag types of the increased metadata information is obtained as follows:

[0121]

[0122] In implementation, the system of the present invention increases the number of metadata information tag types according to the loss ratio of backup data by setting a preset loss ratio. Due to the long-term resource limitation of the system, there are omissions in the types of metadata information tagged, lacking types such as data lifecycle tags and metadata keyword tags, resulting in the inability to accurately manage and track data during archiving and deletion, thereby causing data loss and insufficient comprehensiveness of metadata information tags. The number of metadata information types is increased to reduce omissions in the types of metadata information tagged, thereby improving the comprehensiveness of metadata information tags and further improving the effectiveness of data backup.

[0123] The present invention also provides a data backup and disaster recovery method of a data backup and disaster recovery system based on a data lake platform, comprising:

[0124] Step S1, regularly backing up the data in the data lake;

[0125] Step S2, creating several versions of the data lake and marking the metadata information in each version;

[0126] Step S3, when a disaster occurs, the backed-up data and the marked metadata information are restored, and the data backup process and the disaster recovery process are monitored in real time;

[0127] Step S4, respectively obtaining the restoration duration of several backup data of the same data volume and the total number of times the backup data is restored;

[0128] Step S5, determining whether the validity of the data backup meets the requirements according to the average recovery time of the backup data;

[0129] Step S6, if it is determined that the validity does not meet the requirement, then adjusting the data backup method or performing a determination function on the validity of the mark of the metadata information in combination with the data difference byte ratio of the metadata information;

[0130] Step S7, determining a response to the marking question according to the determination result of the marking validity;

[0131] The marking problem response includes adjusting an early warning trigger threshold or adjusting the number of marking types of metadata information according to the proportion of backup data loss.

[0132] So far, the technical method of the present invention has been described in conjunction with the preferred embodiments shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical methods after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A data backup and disaster recovery system based on a data lake platform, characterized in that: include: Data backup module, used to back up all data in the data lake platform; A snapshot module, which is connected to the data backup module and is used to generate a corresponding snapshot when the data in the data lake changes, including a version creation component for creating several versions of the data lake and a marking component connected to the version creation component for marking metadata information in each version; A disaster recovery module, which is connected to the data backup module and the snapshot module respectively, and is used to restore the backup data and the snapshot; A control module is connected to the data backup module, the snapshot module and the disaster recovery module respectively, and is used to determine whether to issue an information warning according to whether an abnormality occurs in the data backup process and the disaster recovery process, and, when it is determined that the validity of the data backup does not meet the requirements according to the average time of the backup data recovery, adjust the data backup method or perform a determination function on the validity of the mark of the metadata information in combination with the data difference byte ratio of the metadata information, and determine a mark problem response according to the determination result of the validity of the mark, The marking problem response includes adjusting the warning trigger threshold or adjusting the number of marking types of metadata information according to the loss ratio of backup data, and the marking types of metadata information include data recovery time, data change record and data change time; The control module obtains the recovery time of several backup data with the same data volume and the total number of backup data recovery times, and calculates the average recovery time of the backup data according to the obtained results. The control module determines that the validity of the data backup does not meet the requirement under the preset first average duration condition or the preset second average duration condition; The control module adjusts the sequential backup mode to the priority backup mode when only the preset first time period is met; Among them, the preset first average duration condition is that the average duration of the backup data recovery is greater than the preset second average duration; the preset second average duration condition is that the average duration of the backup data recovery is greater than the preset first average duration and less than or equal to the preset second average duration; The control module preliminarily determines that the marking validity of the metadata information does not meet the requirements under the preset second average duration condition, and performs a determination function for the marking validity of the metadata information in combination with the data difference byte ratio of the metadata information, including: The data bytes that are inconsistent between the marked metadata information and the data in the data lake platform and the total bytes of the marked metadata information are obtained respectively, and the proportion of the data difference bytes of the metadata information is calculated based on the obtained results, where: The control module secondarily determines that the marking validity of the metadata information does not meet the requirement under a preset first difference byte ratio condition or a preset second difference byte ratio condition, and executes the marking problem response; The control module reduces the warning trigger threshold when only the preset first difference byte ratio is met. Among them, the preset first difference byte ratio condition is that the data difference byte ratio of the metadata information is greater than the preset second difference byte ratio; the preset second difference byte ratio condition is that the data difference byte ratio of the metadata information is greater than the preset first difference byte ratio and less than or equal to the preset second difference byte ratio; The control module preliminarily determines that the comprehensiveness of the metadata information mark does not meet the requirement under the preset second difference byte amount ratio condition, and obtains the lost byte amount of the backup data and the total byte amount of the backup data respectively, and calculates the lost amount ratio of the backup data according to the obtained results. The control module determines for the second time that the comprehensiveness of the metadata information mark does not meet the requirement under the preset loss amount ratio condition, and increases the number of the mark types of the metadata information; The preset loss amount ratio condition is that the loss amount ratio of the backup data is greater than the preset loss amount ratio.

2. The data backup and disaster recovery system based on the data lake platform according to claim 1, characterized in that: The data backup module includes: A capture component is used to capture the changed data in all data in the data lake platform; The backup storage component is connected to the capture component and is used to store the captured data.

3. The data backup and disaster recovery system based on the data lake platform according to claim 1, characterized in that: The sequential backup is to back up all the data on the data lake platform in order of data volume, and the priority backup is to back up all the data on the data lake platform according to the priority of the data.

4. The data backup and disaster recovery system based on the data lake platform according to claim 1, characterized in that: The calculation formula for the average duration of restoring the backup data is: Among them, G is the average time for restoring backup data, H x is the duration of restoring the x-th backup data, and y is the total number of times the backup data is restored, wherein the amount of data in each backup is the same.

5. The data backup and disaster recovery system based on the data lake platform according to claim 1, characterized in that: The reduction range of the early warning triggering threshold is determined by the difference between the data difference byte ratio of the metadata information and the preset second difference byte ratio.

6. The data backup and disaster recovery system based on the data lake platform according to claim 1, characterized in that: The increase in the number of the marking types of the metadata information is determined by the difference between the missing amount ratio of the backup data and the preset loss amount ratio.

7. A data backup and disaster recovery method for the data backup and disaster recovery system based on a data lake platform as claimed in any one of claims 1 to 6, characterized in that: The following steps are involved: Regularly back up the data in the data lake; Creating several versions of the data lake and marking metadata information in each version; When a disaster occurs, the backed-up data and marked metadata information are restored, and the data backup process and disaster recovery process are monitored in real time; Obtain the recovery time of several backup data with the same data volume and the total number of backup data recovery times; Determining whether the validity of the data backup meets the requirements based on the average recovery time of the backup data; If it is determined that the validity does not meet the requirements, the data backup method is adjusted or the determination function of the marking validity of the metadata information is performed in combination with the data difference byte ratio of the metadata information; Determining a response to a marking question based on the determination result of the marking validity; The marking problem response includes adjusting an early warning trigger threshold or adjusting the number of marking types of metadata information according to the proportion of backup data loss.

Images