A database backup method

By analyzing the characteristics and complexity of database backup data, combining Spark parallel computing and transaction consistency control, an automated backup strategy is adopted to solve the performance bottlenecks and consistency problems of traditional backup methods, and efficient and reliable database backup is achieved.

CN118567912BActive Publication Date: 2024-11-08北京科杰科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411034551.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-08
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

Traditional database backup methods have performance bottlenecks, consistency problems and insufficient manual operations, resulting in low backup efficiency, incomplete data and inconvenient management.

Method used

By obtaining the characteristic data of the data to be backed up, analyzing its structural similarity and complexity, it is decided to use full or incremental synchronization to backup, and using Spark's parallel computing and transaction consistency control, combined with automated backup strategies, reduce manual intervention and Redundancy, improving backup efficiency and accuracy.

Benefits of technology

It realizes an efficient, reliable and automated backup process, reduces backup time costs, ensures data consistency, simplifies management processes, and reduces manual errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118567912B_ABST
    Figure CN118567912B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of database backup, and in particular to a database backup method, the method comprising acquiring data to be backed up in a plurality of databases, and extracting characteristic data of the data to be backed up; determining the backup of the data to be backed up based on the structural similarity between the data structure in the characteristic data of the data to be backed up and the structure of the historical backup data structure of a backup engine; determining the analysis of the data to be backed up based on the data complexity of the data to be backed up; determining an execution engine based on the encryption of the data to be backed up to perform integrity verification on the data to be backed up after the backup is completed; determining the qualification of a backup process based on the integrity verification result of the execution engine to determine the adjustment of the backup process based on the unqualified result. The present invention solves the problem of low accuracy of data backup in the backup process of the prior art, resulting in low backup efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of database backup, and in particular to a database backup method. Background Art

[0002] Traditional database backup methods are usually based on single-node backup tools, such as mysqldump or pg_dump, etc. These tools have some problems when backing up large-scale data, including performance bottlenecks: In large-scale data processing scenarios, single-node backup tools may not meet the needs of efficient backup, and the backup process takes a long time, affecting the normal operation of the database. Consistency issues: During the backup process, the database may be in a read-write state. Traditional backup methods may not guarantee the consistency of the backup data, resulting in incomplete or inaccurate backup data. Manual operation: Traditional backup methods usually need to be executed manually and require human intervention. They are prone to errors and are not easy to manage. Especially in automated operation and maintenance scenarios, this manual operation is particularly cumbersome and inefficient. In summary, traditional database backup methods have many shortcomings such as performance bottlenecks, consistency issues and manual operations. A more efficient, reliable and automated backup method is needed to solve these problems.

[0003] Traditional backup methods are inefficient and time-consuming when backing up large amounts of data, and cannot meet the needs of real-time backup. Therefore, an efficient backup method is needed to speed up the backup process and reduce the time cost of backup. Traditional backup methods cannot guarantee the consistency of backup data during the backup process, which may result in incomplete or inaccurate backup data. Therefore, a backup method is needed that can ensure the consistency of backup data when the database is in a read-write state. Traditional backup methods usually need to be executed manually, require human intervention, are prone to errors, and are not easy to manage. Therefore, an automated backup method is needed that can reduce human intervention, simplify the backup management process, and improve the reliability and consistency of backup. Summary of the invention

[0004] To this end, the present invention provides a database backup method to overcome the problems in the prior art of database backup, such as performance bottlenecks, consistency issues and manual operations, which result in the process of backing up data not being able to be backed up efficiently, reliably and automatically.

[0005] To achieve the above object, the present invention provides a database backup method, comprising:

[0006] Obtaining data to be backed up from a number of databases, and extracting characteristic data of the data to be backed up;

[0007] Determining the backup of the data to be backed up based on the structural similarity between the data structure in the characteristic data of the data to be backed up and the structural similarity between the historical backup data structure of the backup engine; wherein, determining the backup of the data to be backed up based on the structural similarity between the data structure in the characteristic data of the data to be backed up and the structural similarity between the historical backup data structure of the backup engine includes determining to back up the data to be backed up in a full synchronization manner according to a comparison result in which the structural similarity is less than or equal to a preset structural similarity, or determining to back up the data to be backed up in an incremental synchronization manner according to a comparison result in which the structural similarity is greater than a preset structural similarity;

[0008] Determining analysis of the data to be backed up based on the data complexity of the data to be backed up;

[0009] Determine an execution engine based on the encryption of the data to be backed up to verify the integrity of the data to be backed up after the backup is completed;

[0010] The qualification of the backup process is determined based on the integrity verification result of the execution engine to determine the adjustment of the backup process based on the unqualified result.

[0011] Further, the determining the analysis of the data to be backed up based on the data complexity of the data to be backed up includes determining the eligibility of the data to be backed up based on unstructured data in the non-encrypted data of the backup data under the condition that the data complexity is less than or equal to a preset complexity.

[0012] Further, the determining the analysis of the data to be backed up based on the data complexity of the data to be backed up includes determining the eligibility of the data to be backed up based on the changed data in the data to be backed up under the condition that the data complexity is greater than a preset complexity.

[0013] Furthermore, the integrity verification of the data to be backed up that has been backed up includes executing the data to be backed up through an execution engine, and based on the condition that encrypted data exists in the data to be backed up, only executing non-encrypted data segments with complete data language, and determining that the ratio of the data volume of the data segment to the total data volume of the data to be backed up is a first preset ratio.

[0014] Furthermore, the integrity verification of the data to be backed up that has been backed up includes executing the data to be backed up through an execution engine, and based on the condition that there is no encrypted data in the data to be backed up, executing a data segment with a complete data language and the ratio of the data volume of the data segment to the total data volume of the data to be backed up is a second preset ratio.

[0015] Furthermore, determining the eligibility of the backup process based on the integrity verification result of the execution engine includes determining the execution results after the execution engine executes several of the data segments, and determining whether a single execution result is completed, and determining that the data to be backed up is incomplete based on the difference between the completion rate and the preset completion rate under the condition that the completion rate of the execution results of several of the data segments is less than a preset completion rate.

[0016] Further, the determining of the adjustment of the backup process based on the unqualified result includes adjusting the preset data similarity by an adjustment coefficient under the condition that the difference is less than or equal to a preset difference.

[0017] Furthermore, the adjustment of the backup process based on the unqualified result includes compensating the preset structural similarity with a compensation coefficient under the condition that the difference is greater than a preset difference.

[0018] Compared with the prior art, the beneficial effect of the present invention lies in that the present invention determines the backup method of the data to be backed up by performing a similarity analysis on the data to be backed up and the historical backup data, so that the backup engine can perform targeted backup when performing the backup. For example, when the database updates data, the backup of the data to be backed up is performed in an incremental synchronization manner, and when the database performs a new data backup, the backup of the data to be backed up is performed in a full synchronization manner, thereby realizing flexible control of the backup engine, improving the accuracy of the backup engine's data backup, reducing the redundancy of the backup data, and thereby improving the efficiency of the backup.

[0019] Furthermore, the present invention determines the data complexity of the data to be backed up, executes different analysis methods on the data to be backed up under different complexities, and analyzes part of the backup data under the corresponding backup method, thereby realizing flexible control of the backup process, improving the accuracy of data backup of the backup engine, reducing the redundancy of backup data, and thereby improving the efficiency of backup.

[0020] Furthermore, the present invention selects several unencrypted data segments for the execution engine to execute, thereby ensuring the security of the user's backup data, and determines the execution result of each segment of data. Based on a copy of the data to be backed up, if several data segments have execution results, it determines whether the execution result of each segment of data is completed, and determines the completeness of the data to be backed up after backup based on the completion rate of the execution results of several data segments, thereby achieving precise control of the backup process of the backup data, thereby improving the efficiency of backup.

[0021] Furthermore, the present invention determines the adjustment of the backup process according to the difference in completion rate when it is determined that the data to be backed up is incomplete, and selects different adjustment methods based on different differences, so as to achieve precise control of the backup process of the backup data, thereby improving the backup efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a database backup method according to an embodiment of the present invention;

[0023] Figure 2 A flowchart of determining a backup mode in a database backup method according to an embodiment of the present invention;

[0024] Figure 3 A flowchart of determining an analysis method in a database backup method according to an embodiment of the present invention;

[0025] Figure 4 The present invention is a flowchart of the integrity verification in the database backup method according to the embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0027] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0028] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0029] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0030] The database backup method of the present invention adopts the following technical solutions:

[0031] Spark technology: This invention uses Spark as the backup engine, taking advantage of its distributed computing and memory computing to realize the parallel backup process, greatly improving the backup efficiency. Spark can decompose the backup task into multiple subtasks and execute them in parallel on multiple nodes in the cluster, thereby speeding up the backup process and reducing the backup time cost.

[0032] Transaction consistency control: In order to ensure the consistency of backup data, the present invention adopts a transaction consistency control mechanism. During the backup process, the present invention will identify the ongoing transaction based on the transaction log of the database, and set the backup point after the transaction is committed, thereby ensuring the consistency of the backup data. At the same time, the present invention also supports incremental backup, only backing up the changed data, reducing the redundancy of the backup data and improving the backup efficiency.

[0033] Automated backup strategy: In order to realize the automation of backup, the present invention provides a flexible backup strategy configuration. Users can configure parameters such as backup cycle, backup time point and backup storage location according to their own needs. The system will automatically perform backup tasks according to the configuration, reducing manual intervention and simplifying the backup management process.

[0034] In summary, the present invention adopts Spark-based parallel backup technology, transaction consistency control mechanism, and automated backup strategy, which solves the cumbersome and inefficient problems of traditional backup methods, improves backup efficiency, ensures backup consistency, and realizes automated backup management.

[0035] The database backup method of the present invention utilizes the parallel computing capability of Spark to accelerate the backup process and reduce the backup time cost. The automated backup process reduces human errors and improves the consistency and reliability of backup. It reduces manual intervention, simplifies the backup management process, and reduces management costs.

[0036] See also Figure 1-Figure 3 As shown, Figure 1 A flowchart of a database backup method according to an embodiment of the present invention; Figure 2 A flowchart of determining a backup mode in a database backup method according to an embodiment of the present invention; Figure 3 A flowchart of determining an analysis method in a database backup method according to an embodiment of the present invention; Figure 4 The present invention is a flowchart of the integrity verification in the database backup method according to the embodiment of the present invention.

[0037] The data backup method according to the embodiment of the present invention includes:

[0038] Step S1, obtaining data to be backed up from a number of databases, and extracting characteristic data of the data to be backed up;

[0039] Step S2: determining a backup method for the data to be backed up based on the structural similarity between the data structure in the characteristic data of the data to be backed up and the historical backup data structure of the backup engine;

[0040] Step S3, determining an analysis method for the data to be backed up under a corresponding backup method based on the data complexity of the data to be backed up;

[0041] Step S4: determining an execution engine based on the encryption of the data to be backed up, so as to verify the integrity of the data to be backed up after the backup is completed;

[0042] Step S5: determining the eligibility of the backup process based on the integrity verification result of the execution engine, and determining the adjustment of the backup process based on the unqualified result.

[0043] Specifically, when determining the backup method for the data to be backed up, the data structure of the data to be backed up is compared with the historical backup data structure of the backup engine according to the structural similarity and the preset structural similarity, so as to determine the backup method for the data to be backed up based on the comparison result;

[0044] If the structural similarity is less than or equal to a preset structural similarity, determining to back up the data to be backed up in a first backup mode;

[0045] If the structural similarity is greater than a preset structural similarity, it is determined to back up the data to be backed up in a second backup manner.

[0046] In the embodiment of the present invention, the first backup mode includes backing up the data to be backed up in a full synchronization mode, and the second backup mode includes backing up the data to be backed up in an incremental synchronization mode.

[0047] In an embodiment of the present invention, the preset similarity value is 0.6, that is, when the data structure of the data to be backed up and the historical backup data has a 60% similarity structure, the corresponding backup method is selected to perform the backup of the data to be backed up, but the above value is not limited to this, and technical personnel in this field can also adjust the value according to actual conditions.

[0048] Specifically, the present invention performs a similarity analysis between the data to be backed up and the historical backup data to determine the backup method for the data to be backed up, so that the backup engine can perform targeted backup when performing backup. For example, when the database updates data, the backup of the data to be backed up is performed in an incremental synchronization manner, and when the database performs a new data backup, the backup of the data to be backed up is performed in a full synchronization manner, thereby achieving flexible control of the backup engine, improving the accuracy of the backup engine's data backup, reducing the redundancy of the backup data, and thereby improving the efficiency of the backup.

[0049] Specifically, when determining the analysis method of the data to be backed up under the corresponding backup method, the analysis method is determined according to the comparison result of the data complexity of the data to be backed up and the preset complexity;

[0050] If the data complexity is less than or equal to the preset complexity, determining to analyze the data to be backed up in a first analysis manner;

[0051] If the data complexity is greater than a preset complexity, a second analysis method is determined to analyze the data to be backed up.

[0052] In the embodiment of the present invention, the preset complexity value is 0.85, but the above value is not limited thereto, and those skilled in the art may also adjust the value according to actual conditions.

[0053] Specifically, when the data to be backed up is analyzed in a first analysis manner, whether the data to be backed up is qualified is determined based on a comparison result of the data similarity between the unstructured data in the unencrypted data of the backup data and the unstructured data in the unencrypted data of the historical backup data and a preset data similarity;

[0054] If the data similarity is less than or equal to the preset data similarity, determining that the data to be backed up is unqualified;

[0055] If the data similarity is greater than a preset data similarity, it is determined that the data to be backed up is qualified.

[0056] In the embodiment of the present invention, the preset data similarity value of the unstructured data is 0.9, but the above value is not limited thereto, and those skilled in the art may also adjust the value according to actual conditions.

[0057] Specifically, when the data to be backed up is analyzed in the second analysis mode, the eligibility of the data to be backed up is determined based on the data similarity between the changed data in the data to be backed up and the historical backup data;

[0058] If the data similarity is less than or equal to the preset data similarity, determining that the data to be backed up is unqualified;

[0059] If the data similarity is greater than a preset data similarity, it is determined that the data to be backed up is qualified.

[0060] In the embodiment of the present invention, the preset data similarity value between the change data and the historical backup data is 0.5, but the above value is not limited thereto, and those skilled in the art may also adjust the value according to actual conditions.

[0061] Specifically, the present invention determines the data complexity of the data to be backed up, executes different analysis methods on the data to be backed up under different complexities, and analyzes part of the backup data under the corresponding backup method, thereby realizing flexible control of the backup process, improving the accuracy of the backup engine's data backup, reducing the redundancy of the backup data, and thus improving the efficiency of the backup.

[0062] Specifically, when the integrity of the data to be backed up is verified, the data to be backed up is executed by the execution engine, and the execution mode of the execution engine to execute the data to be backed up is determined based on whether there is encrypted data in the data to be backed up;

[0063] If the data to be backed up contains encrypted data, only the data segment that is not encrypted and has a complete data language is executed, and the ratio of the data volume of the data segment to the total data volume of the data to be backed up is determined to be a first preset ratio;

[0064] If the data to be backed up does not contain encrypted data, a data segment with a complete data language is executed and the ratio of the data volume of the data segment to the total data volume of the data to be backed up is a second preset ratio.

[0065] In the embodiment of the present invention, the first preset ratio is smaller than the second preset ratio, the value of the first preset ratio is 0.2, and the value of the second preset ratio is 0.5.

[0066] Specifically, when determining the eligibility of the backup process based on the integrity verification result of the execution engine, determining the execution result after the execution engine executes the plurality of the data segments, and determining whether a single execution result is completed, and determining whether the data to be backed up is complete based on the comparison result of the completion rate of the execution results of the plurality of the data segments with the preset completion rate;

[0067] If the completion rate is greater than or equal to a preset completion rate, it is determined that the data to be backed up is complete;

[0068] If the completion rate is less than a preset completion rate, it is determined that the data to be backed up is incomplete.

[0069] In the embodiment of the present invention, the preset completion rate is set to 0.95, but the value is not limited thereto, and those skilled in the art may also adjust the value according to actual conditions.

[0070] Specifically, the present invention selects several unencrypted data segments for the execution engine to execute, thereby ensuring the security of the user's backup data, and determines the execution result of each segment of data. Based on a copy of the data to be backed up, if several data segments have execution results, it determines whether the execution result of each segment of data is completed, and determines the completeness of the data to be backed up after backup based on the completion rate of the execution results of several data segments, thereby achieving precise control of the backup process of the backup data, thereby improving the efficiency of backup.

[0071] Specifically, when it is determined that the data to be backed up is incomplete, a difference between the completion rate and a preset completion rate is calculated, so as to determine an adjustment method for the backup process according to a comparison result between the difference and the preset difference;

[0072] If the difference is less than or equal to a preset difference, determining to adjust the backup process in a first adjustment manner;

[0073] If the difference is greater than a preset difference, it is determined to adjust the backup process in a second adjustment manner.

[0074] In the embodiment of the present invention, the value of the preset difference is 0.3, but the above value is not limited to this, and those skilled in the art can also adjust the value according to actual conditions.

[0075] Specifically, when it is determined to adjust the backup process in the first adjustment manner, the preset data similarity is adjusted by the adjustment coefficient.

[0076] In the embodiment of the present invention, the adjustment coefficient has a value range of 1.1-1.15, and a preferred value is 1.18.

[0077] Specifically, when it is determined to adjust the backup process in the second adjustment manner, the preset structural similarity is compensated with a compensation coefficient.

[0078] In the embodiment of the present invention, the compensation coefficient has a value range of 1.2-1.3, and a preferred value is 1.25.

[0079] Specifically, the present invention determines the adjustment of the backup process according to the difference in completion rate when it is determined that the data to be backed up is incomplete, and selects different adjustment methods based on different differences, so as to achieve precise control of the backup process of the backup data, thereby improving the backup efficiency.

[0080] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A database backup method, characterized in that: include: Obtaining data to be backed up from a number of databases, and extracting characteristic data of the data to be backed up; Determining the backup of the data to be backed up based on the structural similarity between the data structure in the characteristic data of the data to be backed up and the historical backup data structure of the backup engine; Determine, based on the data complexity of the data to be backed up, to analyze the unstructured data in the non-encrypted data of the data to be backed up or the changed data in the data to be backed up; Determine an execution engine based on the encryption of the data to be backed up to verify the integrity of the data to be backed up after the backup is completed; Determine the eligibility of the backup process based on the integrity verification result of the execution engine, and determine to adjust the data similarity parameters of the unstructured data in the non-encrypted data and the unstructured data in the non-encrypted data of the historical backup data based on the unqualified result, or compensate the structural similarity parameters of the data structure in the characteristic data of the data to be backed up and the historical backup data structure of the backup engine, so as to determine the adjustment of the backup process; The determining of backing up the data to be backed up based on the structural similarity between the data structure in the characteristic data of the data to be backed up and the historical backup data structure of the backup engine includes determining to back up the data to be backed up in a full synchronization manner according to a comparison result that the structural similarity is less than or equal to a preset structural similarity; The determining of backing up the data to be backed up based on the structural similarity between the data structure in the characteristic data of the data to be backed up and the historical backup data structure of the backup engine includes determining to back up the data to be backed up in an incremental synchronization manner according to a comparison result in which the structural similarity is greater than a preset structural similarity.

2. The database backup method according to claim 1, characterized in that: The determining of analyzing the data to be backed up based on the data complexity of the data to be backed up includes determining the eligibility of the data to be backed up based on unstructured data in the non-encrypted data of the backup data under the condition that the data complexity is less than or equal to a preset complexity.

3. The database backup method according to claim 1, characterized in that: The determining of analyzing the data to be backed up based on the data complexity of the data to be backed up includes determining the eligibility of the data to be backed up based on the changed data in the data to be backed up under the condition that the data complexity is greater than a preset complexity.

4. The database backup method according to claim 3, characterized in that: The integrity verification of the data to be backed up that has been backed up includes executing the data to be backed up through an execution engine, and based on the condition that encrypted data exists in the data to be backed up, only executing non-encrypted data segments that exist in complete data language, and determining that the ratio of the data volume of the data segment to the total data volume of the data to be backed up is a first preset ratio.

5. The database backup method according to claim 3, characterized in that: The integrity verification of the data to be backed up that has been backed up includes executing the data to be backed up through an execution engine, and based on the condition that there is no encrypted data in the data to be backed up, executing a data segment with a complete data language and the ratio of the data volume of the data segment to the total data volume of the data to be backed up is a second preset ratio.

6. The database backup method according to claim 4 or 5, characterized in that: The determination of the eligibility of the backup process based on the integrity verification result of the execution engine includes determining the execution result after the execution engine executes several of the data segments, determining whether a single execution result is completed, and determining that the data to be backed up is incomplete under the condition that the completion rate of the execution results of several of the data segments is less than a preset completion rate.

7. The database backup method according to claim 6, characterized in that: The adjustment of the backup process based on the unqualified result includes calculating the difference between the completion rate and the preset completion rate, and adjusting the preset data similarity with an adjustment coefficient under the condition that the difference is less than or equal to the preset difference.

8. The database backup method according to claim 7, characterized in that: The adjusting of the backup process based on the unqualified result includes compensating the preset structural similarity with a compensation coefficient under the condition that the difference is greater than a preset difference.