A multi-node backup system and method for a database

Abstract

A multi-node backup system for a database, by proposing a method and system for multi-machine and multi-thread backup of structured data collected by a master data system, is used for concurrent backup of structured data, improving the concurrent processing and response capabilities of the master data system. The invention can be adapted to various structured databases such as ChinaDB, TencentDB, and DM Database. The present invention can split the backup task of a master data system into multiple subtasks and hand them over to multiple computers outside the master data system for simultaneous execution. Each backup computer can be further divided into multiple backup threads. By means of simultaneous cooperation and backup of multiple processes and multiple threads, the present invention can improve the backup speed of structured data in the master data system without affecting the response capabilities of the master data system.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and in particular relates to a multi-node backup system and method for a database. Background Art

[0002] In a master data system, a large amount of data, even a huge amount of data, will be collected and managed. A structured database is a common way to store and use this data, and the data in the database involves database backup operations during migration, backup, and restoration. In traditional database backup modes, most are based on backup tools provided by the database itself, and there are also backup tools provided by third parties. When using these tools to perform database backup work, each single backup operation is executed on a single machine of the backup-initiated computer. When the amount of data in the database is large, for example, when the database capacity reaches hundreds of gigabytes or even is calculated in terabytes, using the traditional backup mode, on the one hand, the backup time consumption is large, and the execution of the backup task takes several hours or even longer; on the other hand, the resource requirements for the backup machine itself are also high, such as long-term occupation of CPU and memory resources, resulting in backup interruption or even system crash. In a master data management system, various emergencies are inevitable. When facing an emergency and needing to quickly and reliably backup a large amount of data, using the traditional backup method cannot meet the requirements of emergency backup, thus causing a data disaster. Summary of the Invention

[0003] Object of the Invention: The object of the present invention is to provide a multi-node backup system for a database with multi-node and multi-thread backup and fast backup speed. Another object of the present invention is to provide a multi-node backup method for a database.

[0004] Technical Solution: A multi-node backup system for a database according to the present invention includes a master data platform database and at least one backup node for backing up the data stored in the master data platform database. The backup node includes a system resource statistics module, a node communication management module, a database connection module, a database information statistics module, a backup task splitting module, a backup task execution module, and a backup result feedback module. The system resource statistics module is used to count the resources and capabilities of the participating backup nodes. The node communication management module distributes the backup tasks to each backup node, and after receiving the notification of successful backup messages from all nodes, this module performs file transfer to obtain the backup results of each backup node. The database connection module is used to manage the connection between the backup node and the master data platform database and save the connection configuration between the two. The database information statistics module is used to count the basic information of the master data platform database to be backed up. The backup task splitting module is used for the division of backup tasks. The backup task execution module is used for data backup, and the backup result feedback module is used to notify the backup node to feedback the backup result file.

[0005] Among them, the information node information statistically analyzed by the system resource statistics module includes the available hard disk space, network bandwidth, processing speed, and memory size of each node.

[0006] Among them, the database information statistics module is used to obtain the basic information of the main data platform database, and the basic information includes the type of the database, the current version, the number of data tables, the data dictionary of each table, the number of records in each table, the dependencies between data tables, views, functions, and stored procedures.

[0007] Among them, when the backup task splitting module divides tasks, it will first count the database objects with dependencies, regard the objects with dependencies on each other as a backup unit, and count the data volume of the whole unit; for data objects without dependencies, they are regarded as a backup unit respectively; during the task allocation process, multiple objects with dependencies on each other, as a whole backup unit, will obtain a higher backup sorting priority and be placed in the front part of the backup queue, and the remaining data objects without dependencies will be sorted in the queue according to the data volume; when allocating tasks, the backup task division strategy module will first take out the backup unit from the head of the sorted queue and preferentially allocate it to the node with sufficient resources. After the allocation is completed, it will take the next unit in the queue and continue to allocate until it is completed; during the allocation process, if the backup capacity of a node is insufficient, for example, the data volume of the unit to be backed up exceeds the storage space of the backup node, the task allocation for this node will be skipped; after the task division is completed, the initiating node will send the result of the task allocation to each backup node through the node communication management module.

[0008] Among them, when the backup task execution module executes the allocated tasks for objects with dependencies, it will back up them one by one according to the dependencies. At this time, the backup is performed in a single-threaded manner to ensure the standardization of the backup results; for objects without dependencies, the backup is performed in a single-threaded or multi-threaded manner according to the capabilities of the backup nodes themselves.

[0009] A multi-node backup method for a database, characterized by including the following steps:

[0010] Step 1: Establish a communication connection between backup nodes to obtain the resource information of the backup nodes;

[0011] Step 2: Divide tasks according to the resource information of the backup nodes and the main data platform database;

[0012] Step 3: After receiving the backup task, the backup node executes the task and feeds back the backup result.

[0013] Among them, in Step 1, establishing a communication connection between backup nodes to obtain the resource information of the backup nodes includes the following steps:

[0014] Step 11: Designate any one of the backup nodes as the configuration node. The configuration node starts a listening port to receive requests from other backup nodes for updating and obtaining configuration information, and allocates storage for receiving and maintaining the resource information of each backup node.

[0015] Step 12: Each backup node obtains the information of the configuration node by receiving the broadcast of the configuration node or by manual designation. After completing the information acquisition, the backup node initiates a connection to the configuration node; each backup node then starts another listening port to receive task requests sent by other backup nodes.

[0016] Step 13: After each backup node starts, it obtains the local resources through the system resource statistics module. Each node periodically statistics the local resources, mainly including network bandwidth, memory size, remaining space of the storage location, and CPU quantity information.

[0017] Step 14: Each node periodically reports its own resource information to the configuration node. The configuration node updates this information into its online node list information, and periodically sends the online node list information to each online node. When a new node joins or an online node exits, the configuration node will promptly send update information to the online nodes.

[0018] Among them, the task division in Step 2 includes the following steps:

[0019] Step 21: The user selects the main data platform database to be backed up in the database connection module on a certain backup node and initiates the backup.

[0020] Step 22: The backup initiation node connects to the main data platform database and queries the database through the database information statistics module to obtain the necessary data information, including object information such as the version, data tables, views, functions, etc. of the database to be backed up, and the dependency relationships between the objects.

[0021] Step 23: After completing the statistics of the information of the database to be backed up, organize this information. Consider the objects with dependency relationships as a backup unit, and statistics the overall data volume of this unit. When backing up, this unit is not split. The remaining data objects without dependency relationships are each regarded as a backup unit.

[0022] Step 24: Organize the task assignment queue. Multiple objects with dependency relationships as a whole backup unit will obtain a higher backup sorting priority and be placed at the front of the backup queue. The remaining data objects without dependencies will be sorted in the queue according to the data volume.

[0023] Step 25. Backup task division for each node. Referring to the backup node table received by this node, start dividing the backup tasks for each node. When allocating tasks, the backup task division strategy module will first take out the backup unit from the head of the sorting queue and preferentially allocate it to the node with more sufficient resources, and then take the next unit from the queue to continue the allocation until completion; during the allocation process, if the backup capacity of a node is insufficient, skip the task allocation for this node.

[0024] Step 26. After the task division is completed, the task initiating node sends a connection request to each backup node. After all connections are established, the allocated tasks are packaged and sent to each backup node.

[0025] Among them, when the backup node executes tasks in step 3, it includes the following steps:

[0026] Step 31. After the backup node receives the backup task, it parses the task content and stores the task unit content in the task queue of this node.

[0027] Step 32. Obtain the task unit from the task queue. The backup units with dependencies will be ranked at the head of the queue. For the units with dependencies, a single-threaded method will be used for backup, and the data objects will be backed up one by one according to the dependencies; after the backup of the dependent objects is completed, for other task units, according to the resource situation of the node itself, a single-threaded or multi-threaded backup mode will be adopted.

[0028] Step 33. Reply the execution result. After the task queue of this node is executed, send a message to the initiating node to report that the task is completed.

[0029] Step 34. Backup result transmission. When all node backup tasks are completed, the backup initiating node sends a result transmission request to each backup node, and each node sends the backed-up result file to the backup initiating node.

[0030] Step 35. Centralized storage of result files. After the backup initiating node receives the backup result files sent back by each node, it stores them in the local folder and performs unified storage, splicing, compression, etc. operations for subsequent data restoration.

[0031] Among them, in step 34, if there is a backup failure in the received message, the backup task is reassigned to the original backup node. If it fails again, a different backup node is used to execute this backup task. After the failure reaches a certain number of times, it is reported for manual processing.

[0032] Beneficial effects: Compared with the prior art, the present invention has the following remarkable improvements: By splitting the backup task of a main data system into multiple subtasks and having multiple computers outside the main data system execute them simultaneously, each backup computer can be further divided into multiple backup threads. Through the collaborative backup method of multiple processes and multiple threads, the present invention can improve the backup speed of structured data in the main data system without affecting the response ability of the main data system. In addition, when the present invention divides and executes tasks, it divides data with and without dependency relationships, ensuring data integrity and further improving the efficiency of data backup. Secondly, after a single backup node completes the task, it will respond to the task requests of other backup nodes to ensure that all data can be stored completely, and any backup node performs the backup of all data, facilitating the data backup of large-capacity databases. Brief Description of the Drawings

[0033] Figure 1 is a schematic structural diagram of the present invention;

[0034] Figure 2 is a schematic diagram of the working process of the present invention. Detailed Embodiments

[0035] As Figure 1 shown, the multi-node backup system of the database in the present invention includes a main data platform database and at least one backup node for backing up the data stored in the main data platform database. The backup node includes a system resource statistics module, a node communication management module, a database connection module, a database information statistics module, a backup task splitting module, a backup task execution module, and a backup result feedback module. Among them, the system resource statistics module is responsible for counting the resources and capabilities of the computers participating in the backup, including the available hard disk space in the backup area, network bandwidth, processing speed, memory size, etc. By counting the quantities of various resources of the computers participating in the backup, its backup ability is quantified, and during the backup execution, tasks are preferentially assigned to the computers with strong backup ability. The node communication management module is used for communication between each node when multiple computers participate in the backup work (here, the computers participating in the backup are called nodes). When the backup service is initiated, the communication module is responsible for distributing the backup tasks to each backup node. After receiving the notification of successful backup messages from all nodes, this module performs file transfer to obtain the backup results of each node.

[0036] In the present invention, one node needs to be selected from each node to become the configuration node. Each node sends the information statistically obtained by the system resource statistics module to the configuration node, and the configuration node updates the node resource information list. Each node can obtain all node resource information through the configuration node. Database connection module, which is used to manage the connection between the node and the database and save the connection configuration from the node to the database. One node can save the connection configurations to multiple databases. When performing backup, just select the corresponding database. This module can support the connection of multiple databases, and the database connection information of each node can be synchronized with each other through the configuration node. Database information statistics module, which is used to statistically obtain the basic information of the target database to be backed up. After the user selects the target database at the backup initiating node, the backup initiating node first establishes a connection with the target database, and then obtains the current basic information of the database, including the type of the database, the current version, the number of data tables, the data dictionary of each table, the number of records in each table, the dependencies between data tables, views, functions, stored procedures, etc. Backup task splitting module is used for the backup initiating node to allocate backup tasks by using the task division strategy module. After the initiating node obtains the information of the target database through the database information statistics module, it combines with the backup capabilities information of each node obtained by the node communication management module to divide the backup tasks. The task division strategy module will first count the database objects with dependencies, regard the objects with dependencies on each other as a backup unit, and count the total data volume of this unit. When backing up, this unit is not split. The remaining data objects without dependencies are regarded as a backup unit respectively. During the task allocation process, multiple objects with dependencies on each other, as a whole backup unit, will obtain a higher backup sorting priority and be placed in the front part of the backup queue. The remaining data objects without dependencies will be sorted in the queue according to the data volume. When allocating tasks, the backup task division strategy module will first take out the backup unit from the head of the sorting queue and preferentially allocate it to the node with sufficient resources. After the allocation is completed, it will take the next unit from the queue and continue the allocation until it is completed. During the allocation process, if the backup capacity of a node is insufficient, for example, the data volume of the unit to be backed up exceeds the storage space of the backup node, the task allocation for this node will be skipped. After the task division is completed, the initiating node sends the result of the task allocation to each backup node through the node communication management module. Backup task execution module is used for each backup node to perform data backup through the backup execution module. After each backup node receives the backup task, it starts to plan and execute the data backup. For objects with dependencies, they will be backed up one by one according to the dependencies. At this time, the backup is performed in a single-threaded manner to ensure the standardization of the backup result. For objects without dependencies, the backup is performed in a single-threaded or multi-threaded manner according to the own capabilities of the backup node. After the backup task is completed, the backup node sends a task completion message to the initiating node.After all backup tasks are completed, the backup initiation node notifies the backup execution node to transmit the backup result file. The backup result file can be centrally stored in a certain directory or merged into a single backup file. The backup result file or file set can be further compressed.

[0037] As Figure 2 shown, the multi-node backup method for the database in the present invention includes the following steps:

[0038] Step 1: Establish a communication connection between backup nodes to obtain the resource information of the backup nodes;

[0039] Step 2: Perform task partitioning according to the resource information of the backup nodes and the main data platform database;

[0040] Step 3: After receiving the backup task, the backup node executes the task and feeds back the backup result.

[0041] Specifically, Step 1 includes: The user needs to first specify a node as the configuration node. This node starts a listening port to receive requests from other nodes for updating and obtaining configuration information, and allocates storage for receiving and maintaining the resource information of each backup node. Each backup node obtains the information of the configuration node by receiving the broadcast of the configuration node or by manual specification. After completing the information acquisition, the backup node initiates a connection to the configuration node. Each backup node will then start a listening port to receive task requests sent by other backup nodes. After each node starts, it obtains the local resources through the resource statistics module. Each node will regularly count the local resources, mainly including information such as network bandwidth, memory size, remaining space of the storage location, and number of CPUs. Each node regularly reports its own resource information to the configuration node. The configuration node updates this information into its online node list information and regularly sends the online node list information to each online node. When a new node joins or an online node exits, the configuration node will promptly send update information to the online nodes.

[0042] Step 2 includes the user selecting the target database to be backed up in the database connection module on a certain backup node and initiating the backup. The backup initiation node will connect to the target database and query the database through the database information statistics module to obtain the necessary data information, including object information such as the version, data tables, views, functions, etc. of the database to be backed up, as well as the dependency relationships between objects. After completing the statistics of the database information to be backed up, these information are sorted out. The objects with dependency relationships are regarded as a backup unit, and the total data volume of this unit is counted. When backing up, this unit is not split. The remaining data objects without dependency relationships are each regarded as a backup unit. The task assignment queue is sorted out. Multiple objects with dependency relationships as a whole backup unit will obtain a higher backup sorting priority and be placed at the front of the backup queue. The remaining data objects without dependencies will be sorted in the queue according to the data volume. The backup tasks of each node are divided. Referring to the backup node table received by this node, the backup tasks of each node are started to be divided. When assigning tasks, the backup task division strategy module will first take out the backup unit from the head of the sorting queue and preferentially assign it to the node with more sufficient resources, and then take the next unit from the queue to continue the assignment until it is completed. During the assignment process, if the backup capacity of a node is insufficient, for example, the data volume of the unit to be backed up exceeds the storage space of the backup node, the task assignment for this node will be skipped. After the task division is completed, the task initiation node sends a connection request to each backup node. After all connections are established, the assigned tasks are packaged and sent to each backup node.

[0043] Step 3 is the backup task execution phase. After receiving the backup task, the backup node parses the task content and stores the task unit content in the task queue of this node. The task unit is obtained from the task queue. The backup units with dependency relationships will be ranked at the head of the queue. For the units with dependency relationships, a single-threaded method will be used for backup, and the data objects will be backed up one by one according to the dependency relationships to ensure the standardization of the backup results. After the backup of the objects with dependency relationships is completed, for other task units, according to the resource situation of the node itself, a single-threaded or multi-threaded backup mode will be adopted to speed up the backup progress. The execution result is replied. After the task queue of this node is executed, a message is sent to the initiation node to report that the task is completed. The backup results are transmitted back. When all node backup tasks are completed, the backup initiation node sends a result back transmission request to each backup node, and each node sends the backup result file to the backup initiation node. The result files are centrally saved; after receiving the backup result files sent back by each node, the backup initiation node stores them in the local folder and performs unified storage, splicing, compression and other operations for subsequent data restoration.

Claims

1. A multi-node backup system for a database, characterized in that, It includes a master data platform database and at least one backup node for backing up the data stored in the master data platform database. The backup node includes a system resource statistics module, a node communication management module, a database connection module, a database information statistics module, a backup task splitting module, a backup task execution module, and a backup result feedback module; The system resource statistics module is used to count the resources and capabilities of the participating backup nodes. The node communication management module distributes the backup tasks to each backup node. After receiving the notification of successful backup messages from all nodes, this module performs file transfer and obtains the backup results of each backup node; The database connection module is used to manage the connection between the backup node and the master data platform database and save the connection configuration between the two. The database information statistics module is used to count the basic information of the master data platform database to be backed up; The backup task splitting module is used for the division of backup tasks, specifically including the following steps: Step 21: The user selects the master data platform database to be backed up in the database connection module on a certain backup node and initiates the backup; Step 22: The backup initiation node will connect to the master data platform database and query the database through the database information statistics module to obtain data information, including the version of the database to be backed up, data tables, views, function object information, and the dependency relationships between objects; Step 23: After completing the statistics of the database information to be backed up, organize this information. Consider the objects with dependency relationships as a backup unit, count the total data volume of this unit, and do not split this unit during backup. The remaining data objects without dependency relationships will be regarded as a backup unit respectively; Step 24: Organize the task assignment queue. Multiple objects with dependency relationships as a whole backup unit will obtain a higher backup sorting priority and be placed at the front of the backup queue. The remaining data objects without dependencies will be sorted in the queue according to the data volume; Step 25: Divide the backup tasks of each node. Refer to the backup node list received by this node and start dividing the backup tasks of each node. During task assignment, the backup task division strategy module will first take out the backup unit from the head of the sorting queue and preferentially assign it to the node with more sufficient resources, and then take the next unit from the queue and continue to assign until completion. During the assignment process, if the backup capacity of a node is insufficient, skip the task assignment to this node; Step 26: After the task division is completed, the task initiation node sends connection requests to each backup node. After all connections are established, the assigned tasks are packaged and sent to each backup node; The backup task execution module is used for data backup, and the backup result feedback module is used to notify the backup node to feedback the backup result file.

2. The multi-node backup system of the database according to claim 1, characterized in that, The information node information counted by the system resource statistics module includes the available hard disk space, network bandwidth, processing speed, and memory size of each node's backup area.

3. The multi-node backup system of the database according to claim 1, characterized in that, The database information statistics module is used to obtain the basic information of the master data platform database, and the basic information includes the type of the database, the current version, the number of data tables, the data dictionary of each table, the number of records in each table, the dependencies between data tables, views, functions, and stored procedures.

4. The multi-node backup system of the database according to claim 1, wherein, When the backup task splitting module divides tasks, it will first count the database objects with dependencies, regard the objects with dependencies on each other as a backup unit, and count the overall data volume of the unit; for data objects without dependencies, they are regarded as individual backup units respectively; During the task allocation process, multiple objects with dependencies on each other, as a whole backup unit, will obtain a higher backup sorting priority and be placed in the front part of the backup queue. The remaining data objects without dependencies will be sorted in the queue according to the size of the data volume; when allocating tasks, the backup task division strategy module will first take out the backup unit from the head of the sorting queue and preferentially allocate it to the node with sufficient resources. After the allocation is completed, it will take the next unit from the queue and continue the allocation until it is completed; during the allocation process, if the backup capacity of a node is insufficient and the data volume of the unit to be backed up exceeds the storage space of the backup node, the task allocation for this node will be skipped; after the task division is completed, the initiating node will send the result of the task allocation to each backup node through the node communication management module.

5. The multi-node backup system of the database according to claim 1, characterized in that, When the backup task execution module executes the allocated tasks, for objects with dependencies, it will back up them one by one according to the dependencies. At this time, the backup is carried out in a single-threaded manner to ensure the standardization of the backup results; for objects without dependencies, the backup execution method, single-threaded or multi-threaded, is determined according to the capabilities of the backup nodes themselves.

6. A backup method for a multi-node backup system applied to the database according to claim 1, characterized in that, It includes the following steps: Step 1: Establish communication connections between backup nodes to obtain the resource information of the backup nodes; Step 2: Perform task division according to the resource information of the backup nodes and the master data platform database; it includes the following steps: Step 21: The user selects the master data platform database to be backed up in the database connection module on a certain backup node and initiates the backup; Step 22: The backup initiating node will connect to the master data platform database and query the database through the database information statistics module to obtain data information, including the version of the database to be backed up, data tables, views, function object information, and the dependency situation between objects; Step 23: After completing the statistics of the information of the database to be backed up, organize this information, regard the objects with dependencies as a backup unit, and count the overall data volume of the unit. The unit will not be split during backup, and the remaining data objects without dependencies will be regarded as individual backup units respectively; Step 24: Organize the task allocation queue. Multiple objects with dependencies on each other, as a whole backup unit, will obtain a higher backup sorting priority and be placed at the front of the backup queue. The remaining data objects without dependencies will be sorted in the queue according to the size of the data volume; Step 25. Backup task division for each node. Referring to the backup node table received by this node, start dividing the backup tasks for each node. When allocating tasks, the backup task division strategy module will first take out the backup unit from the head of the sorting queue and preferentially allocate it to the node with relatively sufficient resources, and then take the next unit from the queue to continue the allocation until completion. During the allocation process, if the backup capacity of a node is insufficient, the task allocation for this node will be skipped. Step 26. After the task division is completed, the task initiating node sends connection requests to each backup node. After all connections are established, the allocated tasks are packaged and sent to each backup node. Step 3. After receiving the backup task, the backup node executes the task and feeds back the backup result.

7. The multi-node backup method of the database according to claim 6, wherein In step 1, to establish the communication connection between backup nodes and obtain the resource information of backup nodes, the following steps are included: Step 11. Designate any one backup node as the configuration node. The configuration node starts a listening port to receive requests from other backup nodes for updating and obtaining configuration information, and opens storage for receiving and maintaining the resource information of each backup node. Step 12. Each backup node obtains the information of the configuration node by receiving the broadcast of the configuration node or by manual designation. After completing the information acquisition, the backup node initiates a connection to the configuration node. Each backup node will then start another listening port to receive task requests sent by other backup nodes. Step 13. After each backup node starts, it obtains the local resources through the system resource statistics module. Each node will regularly count the local resources, mainly including network bandwidth, memory size, remaining space of the storage location, and CPU quantity information. Step 14. Each node regularly reports its own resource information to the configuration node. The configuration node updates this information to its online node list information and regularly sends the online node list information to each online node. When a new node joins or an online node exits, the configuration node will promptly send update information to the online nodes.

8. The multi-node backup method of the database according to claim 6, characterized in that, When the backup node executes the task in step 3, the following steps are included: Step 31. After receiving the backup task, the backup node parses the task content and stores the task unit content in the task queue of this node. Step 32. Obtain the task unit from the task queue. The backup units with dependencies will be ranked at the head of the queue. For the units with dependencies, a single-threaded method will be used for backup, and the data objects will be backed up one by one according to the dependencies. After the backup of the objects with dependencies is completed, for other task units, according to the resource situation of the node itself, a single-threaded or multi-threaded backup mode will be adopted. Step 33. Reply with the execution result. After the task queue of this node is executed, a message is sent to the initiating node to report that the task is completed. Step 34. Backup result transmission. When all node backup tasks are completed, the backup initiating node sends a result transmission request to each backup node, and each node sends the backup result file to the backup initiating node. Step 35: The result files are centrally saved. After the backup initiating node receives the backup result files sent back by each node, it stores them in a local folder and performs unified storage, splicing, and compression operations for subsequent data restoration.

9. The multi-node backup method of the database according to claim 8, wherein In step 34, if there is a backup failure in the received message, the backup task is reassigned to the original backup node. If it fails again, a different backup node is used to execute this backup task. After the failure reaches the preset number of times, it is reported for manual processing.

Images