A MongoDB-based backup method, device, equipment, medium and product

Abstract

Embodiments of the present disclosure disclose a MongoDB-based backup method, device, equipment, medium and product. The method comprises: in the case where it is detected that the backup of a target database is triggered, determining a target node whose synchronization progress with a master node in the target database meets a preset synchronization condition, and configuring the target node as a hidden node; suspending the synchronization task of the hidden node, copying the operation log of the hidden node to obtain a copied target backup file; after the target backup file is divided into backup data blocks through a block transmission mechanism, the backup data blocks are transmitted to the external storage space corresponding to the target database; after determining that all the backup data blocks are transmitted to the external storage space, the node configuration of the hidden node is restored to the configuration of the target node, and the synchronization state of the target node and the master node is restored.

Description

Technical Field

[0001] This disclosure generally relates to the field of database technology, and more particularly to a backup method, apparatus, device, medium, and product based on MongoDB. Background Technology

[0002] MongoDB is a NoSQL (Not Only SQL) database, characterized by high performance, scalability, and flexibility, and is widely used to handle massive amounts of data and complex business logic. Data backup is a crucial step in ensuring data security and business continuity.

[0003] In related technologies, data backup typically relies on third-party tools, which suffers from low backup efficiency and high disk and memory resource consumption. Furthermore, in large-scale data scenarios, the low backup time efficiency may lead to a decline in database service performance. In addition, existing technologies introduce state adjustment commands to sequentially back up the physical files of metadata and basic data to achieve consistent online backup. However, under high load environments, the execution of these commands may suffer from low backup efficiency. Summary of the Invention

[0004] This disclosure addresses some of the shortcomings mentioned in the background art by providing a MongoDB-based backup method, apparatus, device, medium, and product.

[0005] In a first aspect, embodiments of this disclosure provide a MongoDB-based backup method, comprising: If a backup of the target database is detected to be triggered, the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions is identified, and the target node is configured as a hidden node. Pause the synchronization task of the hidden node, copy the operation log of the hidden node, and obtain the copied target backup file; After the target backup file is divided into blocks by a block transfer mechanism, backup data blocks are obtained and the backup data blocks are transferred to the external storage space corresponding to the target database. After confirming that all the backup data blocks have been transferred to the external storage space, the node configuration of the hidden node is restored to the configuration of the target node, and the synchronization state between the target node and the master node is restored.

[0006] In one embodiment of the first aspect, determining a target node whose synchronization progress with the master node in the target database meets preset synchronization conditions includes: Determine the synchronization progress between a specified type of node and the master node in the target database; wherein, the specified type of node includes slave nodes and read-only nodes; The node whose synchronization progress is most similar to that of the master node among the specified node types is identified as the intermediate node; If the synchronization delay between the intermediate node and the master node is less than a preset synchronization delay threshold, the intermediate node is determined to meet the preset synchronization conditions, and the intermediate node is determined to be the target node.

[0007] In one embodiment of the first aspect, the step of dividing the target backup file into blocks using a block transfer mechanism to obtain backup data blocks, and then transferring the backup data blocks to the external storage space corresponding to the target database, includes: The target backup file is compressed to obtain a compressed backup file; The compressed backup file is divided into blocks based on a preset block size to obtain backup data blocks; Based on the data volume of the original data corresponding to the backup data block, the data blocks are sorted from largest to smallest priority to obtain the priority order of the backup data blocks; Based on the priority order, the backup data block is transmitted to the external storage space via multi-threaded transmission.

[0008] In one embodiment of the first aspect, in the event of a failure to transmit any of the backup data blocks, the method further includes: Determine the number of retransmissions for a specified backup data block that failed to transmit; The preset initial retransmission interval is processed based on the number of retransmissions to obtain the specified retransmission interval for the specified backup data block. The specified backup data block is retransmitted based on the specified retransmission interval.

[0009] In one embodiment of the first aspect, each of the backup data blocks is associated with a corresponding identifier; wherein the identifier includes a hash value corresponding to the backup data block; After transferring the backup data block to the external storage space corresponding to the target database, the method further includes: After the backup data blocks are transferred to the external storage space corresponding to the target database, the identification information corresponding to each backup data block is determined; The identification information is concatenated to obtain the concatenated identification information; The concatenated identification information is compared with the global hash value corresponding to the target backup file to obtain the comparison result; If, based on the comparison results, it is determined that the spliced ​​identification information and the global hash value are consistent, then all backup data blocks are transferred to the external storage space.

[0010] In one embodiment of the first aspect, the method further includes: The execution status of all steps in the current backup task chain corresponding to the target database is collected to obtain the execution status information of each step, and the execution status information is stored in the local storage space. When any step fails, rollback and retry operations are performed on that step until the corresponding step is completed or the preset maximum number of retries is reached.

[0011] In one embodiment of the first aspect, the method further includes: If a recovery instruction is detected in the target database, it is determined whether the external storage space meets the data recovery conditions; If the external storage space meets the data recovery conditions, the working tasks of each node in the target database are interrupted according to their importance from low to high. Based on the target backup file corresponding to the target database stored in the external storage space, each node is restored to obtain a restored node, and the restored node is restarted.

[0012] In a second aspect, embodiments of this disclosure provide a MongoDB-based backup device, comprising: The determination module is used to determine the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions when the backup of the target database is detected to be triggered, and to configure the target node as a hidden node. The copy module is used to pause the synchronization task of the hidden node, copy the operation log of the hidden node, and obtain the target backup file after copying. The backup module is used to divide the target backup file into blocks using a block transfer mechanism to obtain backup data blocks, and then transfer the backup data blocks to the external storage space corresponding to the target database. The recovery module is used to restore the node configuration of the hidden node to the configuration of the target node after determining that all the backup data blocks have been transferred to the external storage space, and to restore the synchronization state between the target node and the master node.

[0013] In a third aspect, an electronic device is provided, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform the steps of the first aspect above, or any possible implementation of the first aspect.

[0014] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the first aspect or any possible implementation thereof.

[0015] In a fifth aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method described in the preceding aspects.

[0016] As will be described in detail below, a MongoDB-based backup method, apparatus, device, medium, and product according to embodiments of this disclosure are disclosed. Backup is performed by temporarily configuring a target node whose synchronization progress with the master node meets preset synchronization conditions as a hidden node. After backup is completed, the original configuration of the target node is restored and the synchronization state is restored. By temporarily switching node roles, there is no need for long-term deployment of dedicated hidden nodes, avoiding the long-term occupation of cluster resources by hidden nodes and reducing resource waste. Simultaneously, selecting a node whose synchronization progress matches the master node as the backup source and pausing the synchronization task before copying files ensures that the node data has minimal deviation from the master node and that the data no longer changes, guaranteeing data consistency of the target backup file and preventing data loss.

[0017] File transfer is performed by dividing the target backup file into chunks to obtain backup data blocks, and then transferring these backup data blocks to external storage. This chunked transfer method results in smaller data volumes per transfer, reducing instantaneous disk and network I / O overhead during transmission and mitigating resource contention for database services during backup operations. Furthermore, in case of transmission failures, only the faulty backup data blocks need to be processed, eliminating the need to retransmit the entire backup file, reducing the overhead of repeated transmissions, and minimizing the negative impact of backup operations on database service performance. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a MongoDB-based backup method provided in this embodiment of the disclosure; Figure 2 An architecture diagram of a MongoDB-based backup system provided for embodiments of this disclosure; Figure 3 A schematic diagram of a MongoDB-based backup device provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0019] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present disclosure and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the drawings, not the entire structure.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0021] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0022] Research has shown that MongoDB, a NoSQL (Not Only SQL) database, is characterized by high performance, scalability, and flexibility, and is widely used to handle massive amounts of data and complex business logic. Data backup is a crucial step in ensuring data security and business continuity.

[0023] In related technologies, data backup typically relies on third-party tools, which suffers from low backup efficiency and high disk and memory resource consumption. Furthermore, in large-scale data scenarios, the low backup time efficiency may lead to a decline in database service performance. In addition, existing technologies introduce state adjustment commands to sequentially back up the physical files of metadata and basic data to achieve consistent online backup. However, under high load environments, the execution of these commands may suffer from low backup efficiency.

[0024] Based on the above research, this disclosure provides a MongoDB-based backup method. This method involves temporarily configuring a target node whose synchronization progress with the master node meets preset synchronization conditions as a hidden node. After the backup is completed, the target node's original configuration and synchronization state are restored. By temporarily switching node roles, there is no need for long-term deployment of dedicated hidden nodes, avoiding the long-term occupation of cluster resources by hidden nodes and reducing resource waste. Furthermore, by selecting a node whose synchronization progress matches the master node as the backup source and pausing the synchronization task before copying files, the data on the selected node has minimal deviation from the master node and remains unchanged, ensuring data consistency of the target backup file and preventing data loss.

[0025] File transfer is performed by dividing the target backup file into chunks to obtain backup data blocks, and then transferring these backup data blocks to external storage. This chunked transfer method results in smaller data volumes per transfer, reducing instantaneous disk and network I / O overhead during transmission and mitigating resource contention for database services during backup operations. Furthermore, in case of transmission failures, only the faulty backup data blocks need to be processed, eliminating the need to retransmit the entire backup file, reducing the overhead of repeated transmissions, and minimizing the negative impact of backup operations on database service performance.

[0026] To facilitate understanding of this embodiment, a MongoDB-based backup method disclosed in this disclosure will first be described in detail. The execution entity of the MongoDB-based backup method provided in this disclosure is generally a computer device with certain computing capabilities, such as a terminal device, a server, or other processing devices. In some possible implementations, this MongoDB-based backup method can be implemented by a processor calling computer-readable instructions stored in memory.

[0027] See Figure 1 The diagram shows a flowchart of a MongoDB-based backup method provided in this embodiment of the present disclosure. The method includes steps S101 to S104, wherein: S101. If the backup of the target database is detected to be triggered, determine the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions, and configure the target node as a hidden node.

[0028] In embodiments of this disclosure, a detectable data backup instruction is generated when the target database meets the backup triggering conditions. Upon detecting this data backup instruction, a target node whose synchronization progress with the master node in the target database meets preset synchronization conditions is identified.

[0029] Here, backup triggering conditions include: periodic backup triggering based on a preset backup time window, or backup triggering based on manual user operation, or event-driven backup triggering based on API (Application Programming Interface) calls.

[0030] Here, if the target database meets the backup triggering conditions, a backup task can be created and recorded in the target database. The target database is MongoDB.

[0031] After recording the non-backup task in the target database, resource verification can be performed on the external storage space corresponding to the target database to determine whether the remaining storage space of the external storage space can be used to store the backup data of the target database.

[0032] Here, you can also verify the local storage space corresponding to the target database to determine whether the remaining capacity of the local storage space meets the backup requirements.

[0033] Here, if the remaining capacity of the local storage space is insufficient to meet the backup requirements, the backup task can be delayed, or the expansion process of the target database can be triggered until the remaining capacity of the local storage space meets the backup requirements.

[0034] Here, after determining that the remaining storage space in the external storage space can be used to store backup data of the target database, and that the remaining capacity of the local storage space meets the backup requirements, the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions is identified.

[0035] Here, it can be determined whether hidden nodes exist in the target database. If hidden nodes are found to exist in the target database, it is checked whether the hidden nodes are in an idle state (i.e., no task is being executed).

[0036] When the hidden node is idle, the target database is backed up through the hidden node.

[0037] If it is determined that there are no hidden nodes in the target database, or that the hidden nodes are not in an idle state, then the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions is identified, and the target node is configured as a hidden node.

[0038] S102. Pause the synchronization task of the hidden node, copy the operation log of the hidden node, and obtain the target backup file after copying.

[0039] In the embodiments of this disclosure, the synchronization task of the hidden node can be paused, so that the data file corresponding to the hidden node is completely still and will not change during the pause.

[0040] After pausing the synchronization task of the hidden node, the operation log (oplog) of the hidden node can be copied by directly copying the node data file. This preserves the internal structure of the storage engine data file, reducing resource overhead such as errors caused by data transformation and cursor traversal.

[0041] Here, this disclosure can directly copy the original storage file of the target database without reading data line by line and converting it into JSON / BSON text. There is no CPU or memory conversion overhead, it is fast, and there will be no problems such as data loss or format disorder caused by conversion.

[0042] S103. After dividing the target backup file into blocks using the block transfer mechanism, backup data blocks are obtained and then transferred to the external storage space corresponding to the target database.

[0043] In the embodiments of this disclosure, the target backup file can be divided into blocks by a preset block size to obtain multiple backup data blocks, and the multiple backup data blocks can be stored in the local storage space.

[0044] Subsequently, backup data blocks can be transmitted in parallel using a multi-threaded transmission method to transfer the internal backup data blocks to the external storage space corresponding to the target database.

[0045] Here, the priority of each data backup block can be determined based on preset priority determination conditions. Then, according to the priority of each data backup block, the data backup blocks can be added to the task queue, and the backup data blocks in the task queue can be marked as waiting to be transmitted.

[0046] Then, the backup data blocks in the task queue are transferred to the external storage space corresponding to the target database. During the backup data block transfer process, backup data blocks to be transferred can be continuously added to the task queue until all backup data blocks have been transferred to the external storage space corresponding to the target database.

[0047] S104. After confirming that all backup data blocks have been transferred to external storage space, restore the node configuration of the hidden node to the configuration of the target node, and restore the synchronization state between the target node and the master node.

[0048] In embodiments of this disclosure, after determining that all backup data blocks have been transferred to external storage space, the backup data blocks stored in the local storage space of the target database are deleted.

[0049] After the backup data blocks stored in the local storage space are deleted, the node configuration of the hidden node can be restored, and the oplog synchronization status between the target node and the master node can be restored.

[0050] Here, during the implementation of the above steps, the implementation status of each step is reported after completion, so that if a problem occurs at any step in the task chain, rollback or retry can be performed step by step without having to retry the entire process.

[0051] In the embodiments of this disclosure, firstly, upon detecting that a backup of the target database has been triggered, a target node whose synchronization progress with the master node in the target database meets preset synchronization conditions is identified, and the target node is configured as a hidden node; secondly, the synchronization task of the hidden node is paused, and the operation log of the hidden node is copied to obtain a copied target backup file; thirdly, the target backup file is divided into blocks using a block transfer mechanism to obtain backup data blocks, and the backup data blocks are transferred to the external storage space corresponding to the target database; finally, after confirming that all backup data blocks have been transferred to the external storage space, the node configuration of the hidden node is restored to the configuration of the target node, and the synchronization state between the target node and the master node is restored.

[0052] In the above implementation, backup is performed by temporarily configuring a target node whose synchronization progress matches the master node's preset synchronization conditions as a hidden node. After the backup is completed, the target node's original configuration is restored and the synchronization state is restored. By temporarily switching node roles, there is no need to deploy a dedicated hidden node for a long time, avoiding the hidden node from occupying cluster resources for an extended period and reducing resource waste. Simultaneously, selecting a node whose synchronization progress matches the master node as the backup source and pausing the synchronization task before copying files ensures minimal deviation between the node data and the master node's data, and prevents further data changes, thus guaranteeing data consistency of the target backup file and avoiding data loss.

[0053] File transfer is performed by dividing the target backup file into chunks to obtain backup data blocks, and then transferring these backup data blocks to external storage. This chunked transfer method results in smaller data volumes per transfer, reducing instantaneous disk and network I / O overhead during transmission and mitigating resource contention for database services during backup operations. Furthermore, in case of transmission failures, only the faulty backup data blocks need to be processed, eliminating the need to retransmit the entire backup file, reducing the overhead of repeated transmissions, and minimizing the negative impact of backup operations on database service performance.

[0054] In an optional implementation, the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions is identified, specifically including the following steps: First, determine the synchronization progress between the specified type of nodes and the master node in the target database; the specified type of nodes includes slave nodes and read-only nodes. Secondly, the node with the most similar synchronization progress to the master node among the specified node types is identified as the intermediate node; Finally, if the synchronization delay between the intermediate node and the master node is less than the preset synchronization delay threshold, the intermediate node is determined to meet the preset synchronization conditions and is identified as the target node.

[0055] In embodiments of this disclosure, the synchronization progress between each node in the target database (excluding the master node) and the master node is determined. Where there are no hidden nodes in the target database, the specified types of nodes, excluding the master node, include slave nodes and read-only nodes.

[0056] Here, the nodes of each type in the target database cluster are shown in Table 1 below.

[0057] Table 1

[0058] In this context, each shard node (Mongo-shard) and configuration service node (ConfigServer) of the target database's sharded cluster is typically composed of a cluster of replica sets.

[0059] Here, the synchronization status of each specified type of node can be evaluated based on the synchronization delay of the operation logs of specified type nodes.

[0060] For example, the synchronization delay ΔT of the i-th node of a specified type i oplog The following conditions must be met: △T i oplog =T primary -T i ; Among them, T primary T represents the synchronization status of the master node of the target database. i This represents the synchronization state of the i-th node of a specified type, where i ranges from 1 to n, and n is the number of nodes of the specified type in the target database.

[0061] Here, a node of a specified type corresponding to the minimum synchronization delay can be selected and designated as an intermediate node. If the synchronization delay of this intermediate node is less than a preset synchronization delay threshold (i.e., the maximum allowed synchronization delay), then this intermediate node is designated as the target node.

[0062] For example, the synchronization delay ΔT of intermediate nodes j oplog <Preset synchronization delay threshold τ max The intermediate node is identified as the target node.

[0063] Once the target node is identified, its configuration properties can be modified to set it as a hidden node and lock it to pause synchronization with the master node.

[0064] In an optional implementation, the target backup file is divided into blocks using a chunked transfer mechanism to obtain backup data blocks, which are then transferred to the external storage space corresponding to the target database. This specifically includes the following steps: First, the target backup file is compressed to obtain a compressed backup file; Secondly, the compressed backup file is divided into blocks based on the preset block size to obtain backup data blocks; Secondly, based on the data volume of the original data corresponding to the backup data block, the data blocks are sorted from largest to smallest priority to obtain the priority order of the backup data blocks; Finally, based on priority, the backup data blocks are transferred to the external storage space via multi-threaded transmission.

[0065] In embodiments of this disclosure, the target backup data can be preprocessed to compress the target backup file, resulting in a compressed backup file.

[0066] After the compressed backup file is determined, it can be divided into blocks according to the preset block size (i.e., fixed block size) to obtain backup data blocks.

[0067] Here, those skilled in the art can set the preset block size based on actual needs, for example, setting the preset block size to 5MB.

[0068] Here, the uncompressed data volume of each backup data block can be determined, i.e., the original data volume. Based on the uncompressed data volume of each backup data block, the backup data blocks are prioritized to obtain their priority order.

[0069] Among them, the larger the uncompressed data volume of the backup data block, the higher the priority.

[0070] Here, backup data blocks can be arranged into the task queue based on their priority order, so that each backup data block can be transmitted according to its priority order.

[0071] Here, when the task queue backlog exceeds the threshold, a new thread is added; if the thread's idle time exceeds the maximum space time, the thread resources are released.

[0072] In an optional implementation, in the event of any backup data block transmission failure, the following steps are also included: First, determine the number of retransmissions for the specified backup data block that failed to be transmitted; Secondly, the preset initial retransmission interval is processed based on the number of retransmissions to obtain the specified retransmission interval for the specified backup data block. Finally, the specified backup data blocks are retransmitted based on the specified retransmission interval.

[0073] In embodiments of this disclosure, if the backup data block fails to be transmitted, the backup data block is identified as the designated backup data block.

[0074] If the specified backup data block is being retransmitted for the first time, the specified backup data block will be retransmitted after a preset initial retransmission interval.

[0075] If a specified backup data block is not being retransmitted for the first time, determine the number of retransmissions for that specified backup data block, and determine the retry interval for the next retransmission of that specified backup data block based on the number of retransmissions.

[0076] For example, specify the retry interval T for backup data block k. retry The following conditions must be met: T retry =T base ×2 n（k）-1 ; Where n(k) is the number of retransmissions for the specified backup data block k, and T base The initial retransmission interval is preset. Those skilled in the art can set the preset initial retransmission interval according to actual needs, which will not be elaborated here.

[0077] In the above implementation, the failure blocks (i.e., designated backup data blocks) are handled by an exponential backoff retry mechanism, thereby improving the fault tolerance of block transmission.

[0078] In one optional implementation, each backup data block generates corresponding identification information; wherein, the identification information includes the hash value corresponding to the backup data block.

[0079] After transferring the backup data blocks to the external storage space corresponding to the target database, the following steps are also included: First, after transferring the backup data blocks to the external storage space corresponding to the target database, determine the identification information corresponding to each backup data block; Secondly, the various identification information are spliced ​​together to obtain the spliced ​​identification information; Secondly, the concatenated identification information is compared with the global hash value corresponding to the target backup file to obtain the comparison result; Finally, based on the comparison results, if it is determined that the spliced ​​identification information and the global hash value are consistent, it is determined that all backup data blocks will be transferred to external storage space.

[0080] In embodiments of this disclosure, the identification information also includes a unique identifier corresponding to the backup data block (i.e., the ID of the backup data block).

[0081] After determining that the backup data blocks will be transferred to the external storage space corresponding to the target database, the hash values ​​(i.e., SHA-256 hash values) corresponding to each backup data block can be concatenated to obtain the concatenated hash value.

[0082] Then, the concatenated hash value can be compared with the global hash value corresponding to the target backup file to determine whether the concatenated hash value and the global hash value corresponding to the target backup file are consistent.

[0083] If the hash value after concatenation matches the global hash value corresponding to the target backup file, then all backup data blocks are transferred to external storage space.

[0084] If the hash value after concatenation matches the global hash value corresponding to the target backup file, the missing backup data block is identified based on the difference between the concatenated hash value and the global hash value, and the missing backup data block is retransmitted to ensure the consistency between the backup data block stored in the external storage space and the target backup file.

[0085] In an optional implementation, the following steps are also included: First, the execution status of all steps in the current backup task chain corresponding to the target database is collected to obtain the execution status information of each step, and the execution status information is stored in the local storage space. Secondly, when any step fails, the step is rolled back and retried until the corresponding step is completed or the preset maximum number of retries is reached.

[0086] In the embodiments of this disclosure, when executing the current backup process, the execution status of each process stage under the entire task chain corresponding to this backup is collected in real time, the execution status information of each process stage is generated, and all execution status information is persistently stored in the local storage space.

[0087] Here, the execution status information corresponding to each process stage is continuously monitored. When the execution status of any process stage is marked as failed, the corresponding rollback operation is performed only for the currently failed process stage, and a retry operation is initiated for that process stage. The process stage is repeated until the current backup task is fully completed, or the number of retries reaches the preset maximum number of retries, at which point the current backup task is terminated and an error message is output.

[0088] Here, the rollback operation specifically means undoing all changes performed within the current failed process phase: If the hidden node switching phase fails, the node with the changed role will be restored to its original slave node or read-only node and unlocked; if the block transfer phase fails as a whole, the locally generated temporary backup files and the incomplete backup data blocks uploaded to the external storage space will be cleared; if the node configuration recovery phase fails, the hidden node configuration will be corrected and the data synchronization link between the node and the master node will be restored.

[0089] In the above implementation, after each process stage in the entire task chain is completed, the execution progress, execution result, and error details of that process stage will be collected synchronously as execution status information and saved. The stored execution status information can be read and identified after the backup task is interrupted, and the process stage corresponding to the task stall can be located. When the backup is initiated again, there is no need to repeat the completed process stage. The execution can be directly resumed from the failed process stage, thus realizing breakpoint resume.

[0090] In an optional implementation, the following steps are also included: First, if a recovery command is detected in the target database, determine whether the external storage space meets the data recovery conditions; Secondly, if the external storage space meets the data recovery conditions, the working tasks of each node in the target database are interrupted according to the importance of each node from low to high. Finally, based on the target backup file corresponding to the target database stored in the external storage space, each node is restored to obtain the restored node, and the restored node is restarted.

[0091] In the embodiments of this disclosure, when a recovery instruction is detected for the target database, it is possible to verify whether the target storage path of the external storage space is accessible and to read the metadata, hash identifier information and database version information of the target backup file within the storage path.

[0092] Next, the read information can be matched and verified with the current target database version and global hash value to comprehensively determine whether the external storage space meets the data recovery conditions. At the same time, the remaining local disk storage space of each node of the target database is checked in advance to confirm that the local disk capacity can fully support the decompressed backup data. If the backup file is inaccessible, the file hash or database version does not match, or the local disk space is insufficient, it is determined that the data recovery conditions are not met, the recovery task is marked as failed, and the recovery process is terminated.

[0093] Subsequently, provided that the external storage space meets the data recovery conditions, the working tasks of each node are interrupted sequentially according to the order of importance of the target database nodes from low to high.

[0094] Among them, the importance of nodes, from low to high, is as follows: read-only node, slave node, master node. Before interrupting a node's work tasks, all database write operations are frozen to avoid conflicts between new and old data during the recovery phase. After the node is shut down, the status of each database process is verified to confirm that all nodes are safely offline, preventing the master node from shutting down prematurely and triggering automatic cluster election, thus shortening the overall business interruption time.

[0095] Subsequently, based on the target backup file corresponding to the target database stored in the external storage space, the backup data block can be pulled using a multi-node parallel download method. The multi-node download task is managed by a dynamically scaling thread pool to avoid resource contention caused by multiple nodes downloading at the same time.

[0096] Here, after each backup data block is downloaded, file integrity is verified based on the hash identifier inherent to the data block. If the hash verification is inconsistent, the data block is retransmitted separately. After all backup data blocks have been downloaded, verified, and decompressed, the decompressed original data files are used to overwrite the original data directories on each node, completing the recovery process for each node and obtaining the recovered node.

[0097] In this process, all recovered nodes can be started sequentially in the order of master node, slave node, and read-only node. After all nodes have been started, the synchronization progress of the operation logs of the master and slave nodes in the cluster will be continuously monitored until the cluster data is fully synchronized and the normal read and write business of the database is restored.

[0098] In the above implementation, the recovery process relies on the complete target backup file generated from the previous backup to perform data restoration. It distinguishes the node operation logic between the backup phase and the recovery phase. During the backup phase, a single node is temporarily switched to a hidden node without stopping the node process. During the recovery phase, all nodes are shut down in an orderly manner to avoid data conflicts. Through a layered node shutdown and orderly restart mechanism, the timing of cluster election is effectively controlled, reducing business downtime. Combined with block-based parallel download, single-block hash verification, and failure retransmission mechanisms, the efficiency of multi-node parallel recovery is improved, ensuring the integrity and consistency of cluster data after recovery. At the same time, multiple layers of resource and file verification are added in the pre-recovery phase to intercept abnormal issues such as file corruption, insufficient storage, and version incompatibility in advance, avoiding interruptions during the recovery process and improving the overall robustness of database recovery operations.

[0099] Based on the same inventive concept, this disclosure also provides a MongoDB-based backup system corresponding to the MongoDB-based backup method. Since the principle of the system in this disclosure for solving the problem is similar to the MongoDB-based backup method described above, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be described again.

[0100] Reference Figure 2The diagram shown is an architecture diagram of a MongoDB-based backup system provided in an embodiment of this disclosure. The system includes: a Manager control system 21, a task executor 22, a MongoDB cluster 23, and an external storage system 24; wherein: The Manager control system 21 is used to respond to backup trigger commands, filter target nodes whose synchronization progress meets preset conditions and configure them as hidden nodes, collect and persistently store the execution status of each process stage of the entire task chain, and trigger the rollback and retry of the corresponding stage when the process stage fails.

[0101] The task executors include executor1 and executor2, which communicate bidirectionally with the Manager control system. They are used to copy the data files of hidden nodes, perform compression, chunking, and multi-threaded priority transmission on the target backup file, retransmit backup data blocks that fail to be transmitted according to the exponential backoff interval, and verify the integrity of the backup file based on the chunk hash value.

[0102] The MongoDB cluster includes a replica set architecture cluster 231 and a sharded architecture cluster 232. The replica set architecture cluster has a primary node and secondary nodes, while the sharded architecture cluster has routing nodes (Mongos), shard nodes (Mongo-shard), and configuration service nodes (ConfigServer) to store business data and provide the original physical data files to be synchronized to the hidden nodes.

[0103] External storage system 24 is connected in communication with each task executor to receive and persistently store backup data blocks after they have been transmitted in chunks, and to provide complete backup files to the task executors during the recovery phase.

[0104] This embodiment achieves efficient data backup and recovery based on modular task chain orchestration. It achieves efficient and reliable backup and recovery by executing task chain modules in stages. Specifically, it dynamically manages nodes, switching node roles as needed. Only during backup is the target node with the best synchronization progress dynamically switched to a hidden node, and its original role is immediately restored upon completion, reducing long-term resource occupation. Data files are divided into fixed blocks, and multi-threaded parallel transmission and hash verification improve transmission efficiency and reduce the time spent on a single operation. Furthermore, resource pre-checking and automatic expansion mechanisms prevent interruptions due to insufficient resources during task execution.

[0105] Based on the same inventive concept, this disclosure also provides a MongoDB-based backup device corresponding to the MongoDB-based backup method. Since the principle of the device in this disclosure for solving the problem is similar to the MongoDB-based backup method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0106] Reference Figure 3 The diagram shown is a schematic of a MongoDB-based backup device provided in an embodiment of this disclosure. The device includes: a determination module 31, a replication module 32, a backup module 33, and a recovery module 34; wherein: The determination module is used to determine the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions when the backup of the target database is detected to be triggered, and to configure the target node as a hidden node. The copy module is used to pause the synchronization task of the hidden node, copy the operation log of the hidden node, and obtain the target backup file after copying. The backup module is used to divide the target backup file into blocks using a block transfer mechanism to obtain backup data blocks, and then transfer the backup data blocks to the external storage space corresponding to the target database. The recovery module is used to restore the node configuration of the hidden node to the configuration of the target node after determining that all the backup data blocks have been transferred to the external storage space, and to restore the synchronization state between the target node and the master node.

[0107] This embodiment performs backups by temporarily configuring a target node whose synchronization progress matches the master node's preset synchronization conditions as a hidden node. After the backup is complete, the target node's original configuration is restored, and the synchronization state is restored. By temporarily switching node roles, there is no need for long-term deployment of dedicated hidden nodes, avoiding the long-term occupation of cluster resources by hidden nodes and reducing resource waste. Simultaneously, by selecting a node whose synchronization progress matches the master node as the backup source and pausing the synchronization task before copying files, the node data has minimal deviation from the master node and no further data changes, ensuring data consistency of the target backup file and preventing data loss.

[0108] File transfer is performed by dividing the target backup file into chunks to obtain backup data blocks, and then transferring these backup data blocks to external storage. This chunked transfer method results in smaller data volumes per transfer, reducing instantaneous disk and network I / O overhead during transmission and mitigating resource contention for database services during backup operations. Furthermore, in case of transmission failures, only the faulty backup data blocks need to be processed, eliminating the need to retransmit the entire backup file, reducing the overhead of repeated transmissions, and minimizing the negative impact of backup operations on database service performance.

[0109] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0110] Corresponding to Figure 1In addition to the rate limiting method for interface calls in the present disclosure, this embodiment also provides an electronic device 400, such as... Figure 4 The diagram shown is a structural schematic of an electronic device 400 provided in an embodiment of this disclosure, including: The system includes a processor 41, a memory 42, and a bus 43. The memory 42 stores execution instructions and includes main memory 421 and external memory 422. The main memory 421, also called internal memory, temporarily stores the computational data in the processor 41, as well as data exchanged with external memory such as a hard disk. The processor 41 exchanges data with the external memory 422 through the main memory 421. When the electronic device 400 is running, the processor 41 communicates with the memory 42 through the bus 43, causing the processor 41 to execute the following instructions: If a backup of the target database is detected to be triggered, the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions is identified, and the target node is configured as a hidden node. Pause the synchronization task of the hidden node, copy the operation log of the hidden node, and obtain the copied target backup file; After the target backup file is divided into blocks by a block transfer mechanism, backup data blocks are obtained and the backup data blocks are transferred to the external storage space corresponding to the target database. After confirming that all the backup data blocks have been transferred to the external storage space, the node configuration of the hidden node is restored to the configuration of the target node, and the synchronization state between the target node and the master node is restored.

[0111] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0112] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0113] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0114] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0115] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0116] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0117] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A backup method based on MongoDB, characterized in that, include: If a backup of the target database is detected to be triggered, the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions is identified, and the target node is configured as a hidden node. Pause the synchronization task of the hidden node, copy the operation log of the hidden node, and obtain the copied target backup file; After the target backup file is divided into blocks by a block transfer mechanism, backup data blocks are obtained and the backup data blocks are transferred to the external storage space corresponding to the target database. After confirming that all the backup data blocks have been transferred to the external storage space, the node configuration of the hidden node is restored to the configuration of the target node, and the synchronization state between the target node and the master node is restored.

2. The method as described in claim 1, characterized in that, The process of determining the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions includes: Determine the synchronization progress between a specified type of node and the master node in the target database; wherein, the specified type of node includes slave nodes and read-only nodes; The node whose synchronization progress is most similar to that of the master node among the specified node types is identified as the intermediate node; If the synchronization delay between the intermediate node and the master node is less than a preset synchronization delay threshold, the intermediate node is determined to meet the preset synchronization conditions, and the intermediate node is determined to be the target node.

3. The method as described in claim 1, characterized in that, The step of dividing the target backup file into blocks using a block transfer mechanism to obtain backup data blocks, and then transferring the backup data blocks to the external storage space corresponding to the target database, includes: The target backup file is compressed to obtain a compressed backup file; The compressed backup file is divided into blocks based on a preset block size to obtain backup data blocks; Based on the data volume of the original data corresponding to the backup data block, the data blocks are sorted from largest to smallest priority to obtain the priority order of the backup data blocks; Based on the priority order, the backup data block is transmitted to the external storage space via multi-threaded transmission.

4. The method as described in claim 3, characterized in that, In the event that any of the backup data block transmissions fails, the method further includes: Determine the number of retransmissions for a specified backup data block that failed to transmit; The preset initial retransmission interval is processed based on the number of retransmissions to obtain the specified retransmission interval for the specified backup data block. The specified backup data block is retransmitted based on the specified retransmission interval.

5. The method as described in claim 3, characterized in that, Each of the backup data blocks is assigned a corresponding identifier; wherein, the identifier includes the hash value corresponding to the backup data block; After transferring the backup data block to the external storage space corresponding to the target database, the method further includes: After the backup data blocks are transferred to the external storage space corresponding to the target database, the identification information corresponding to each backup data block is determined; The identification information is concatenated to obtain the concatenated identification information; The concatenated identification information is compared with the global hash value corresponding to the target backup file to obtain the comparison result; If, based on the comparison results, it is determined that the spliced ​​identification information and the global hash value are consistent, then all backup data blocks are transferred to the external storage space.

6. The method as described in claim 1, characterized in that, The method further includes: The execution status of all steps in the current backup task chain corresponding to the target database is collected to obtain the execution status information of each step, and the execution status information is stored in the local storage space. When any step fails, rollback and retry operations are performed on that step until the corresponding step is completed or the preset maximum number of retries is reached.

7. The method as described in claim 1, characterized in that, The method further includes: If a recovery instruction is detected in the target database, it is determined whether the external storage space meets the data recovery conditions; If the external storage space meets the data recovery conditions, the working tasks of each node in the target database are interrupted according to their importance from low to high. Based on the target backup file corresponding to the target database stored in the external storage space, each node is restored to obtain a restored node, and the restored node is restarted.

8. A MongoDB-based backup device, characterized in that, include: The determination module is used to determine the target node whose synchronization progress with the master node in the target database meets the preset synchronization conditions when the backup of the target database is detected to be triggered, and to configure the target node as a hidden node. The copy module is used to pause the synchronization task of the hidden node, copy the operation log of the hidden node, and obtain the target backup file after copying. The backup module is used to divide the target backup file into blocks using a block transfer mechanism to obtain backup data blocks, and then transfer the backup data blocks to the external storage space corresponding to the target database. The recovery module is used to restore the node configuration of the hidden node to the configuration of the target node after determining that all the backup data blocks have been transferred to the external storage space, and to restore the synchronization state between the target node and the master node.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of any one of the MongoDB-based backup methods of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of any one of the MongoDB-based backup methods of claims 1 to 7.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of any one of the MongoDB-based backup methods of claims 1 to 7.

Images