Load-balanced mass file multi-agent concurrent backup system and method
Through the coordination of the main control unit and distributed message queue, the problem of unbalanced proxy allocation in massive file backups is solved, and load balancing and efficient backup of multi-agent backups are realized.
Patent Information
- Application Number
- CN202510514064.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2025-08-05
AI Technical Summary
The existing multi-agent backup solution cannot reasonably allocate backup agents when the massive file directory structure is complex, resulting in unbalanced backups and limited number of agents, which cannot perform best performance.
The combination of main control unit, backup storage, backup agent, coordination module, backup module, recovery module and resource monitoring module is adopted to achieve rapid allocation and load balancing of files through directory scanning threads and distributed message queues, and adjust the backup speed to achieve time consistency between agents.
Load balancing of multi-agent backup jobs is realized, which avoids backup time differences, allocates files reasonably, improves backup efficiency, and is not limited to the increase or decrease of the number of agents, and optimizes backup performance.
Smart Images
Figure CN120429167A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of concurrent backup of massive files, and in particular relates to a load-balanced multi-agent concurrent backup system and method for massive files. Background Art
[0002] With the rapid development of the internet and the advent of the cloud computing era, applications running on file-centric storage infrastructures (web servers, email, electronic records, electronic archives, and digital media) are increasingly representing a growing portion of enterprise data. IDC (International Data Corporation) estimates that 80% of global storage market shipments will be used to store unstructured data, or file data. As this massive volume of file data grows, its backup and recovery becomes a challenge.
[0003] If massive files are backed up using a single backup proxy, even if the single proxy supports multiple channels, it will be limited by the backup bandwidth of the single proxy or the processing performance of the host where the proxy is located, resulting in insufficient backup performance. Therefore, there is a need for concurrent backup using multiple backup proxies. There are roughly two existing multi-proxy backup solutions: one is to establish one or more backup policies for each proxy, and each backup policy backs up a different subdirectory; the other is to scan before the backup to find the number of files in each subdirectory, so that the subdirectories backed up by each proxy contain a similar number of files.
[0004] Existing multi-proxy backup solutions, where each proxy backs up a different subdirectory, are simple to split up the backup source. However, when backing up massive volumes of files with complex directory structures, each proxy can back up an uneven number of files. Some proxies may complete the backup quickly, while others may take too long to complete the backup job. If a pre-backup scan is performed to ensure that each proxy backs up a similar number of files in the subdirectory, this solution will limit the number of proxies in some scenarios. For example, if there are only 10 subdirectories, but two have significantly more files than the other eight, then using two proxies is optimal, which would not maximize the performance of multi-proxy backup. It is impossible to reasonably assign subdirectories to be backed up to each backup agent, and it is difficult to increase or decrease the number of backup agents, which is inflexible. Therefore, it is necessary to develop a new load-balanced massive file multi-agent concurrent backup system and method to solve the existing problems. Summary of the Invention
[0005] The purpose of the present invention is to provide a load-balanced massive file multi-agent concurrent backup system and method to solve the problem that when the massive file directory structure to be backed up is complex, it is impossible to reasonably assign the subdirectory to be backed up to each backup agent.
[0006] To achieve the above objectives, the present invention provides the following technical solution: a load-balanced multi-agent concurrent backup system for massive files, comprising: The main control unit is used to create backup strategies, send backup tasks, manage backup sets, and manage backup jobs; Backup storage, used to store the backup set; at least one backup agent, configured to receive and execute backup tasks from the master control unit and transfer data to the backup storage; The backup agent includes: A backup module, used to transfer the data to be backed up to the backup storage; A coordination module, configured to quickly and non-duplicatively distribute the massive amount of files that need to be backed up to the backup module of each backup agent when multiple agents perform concurrent backups; A recovery module, configured to transfer the data to be recovered from the backup storage to a recovery target location; A resource monitoring module is used to monitor the computing resources of the host where the backup agent is located and adjust the backup and recovery speed during backup and recovery; Among them, the client is installed on the production server.
[0007] Preferably, the coordination module includes: Directory scanning thread, task distributed message queue, and status distributed message queue.
[0008] Preferably, the directory scanning thread scans the massive file directories to be backed up, and stores the scanned files to be backed up in the task distributed message queue. The backup modules of multiple backup agents take out the file messages to be backed up from the message queue for backing up. When the backup agent completes backing up the files, it sends a status update message to the status distributed message queue.
[0009] Preferably, when the number of the backup agents is two or more, one backup agent is selected to start the coordination module.
[0010] Preferably, the main control unit sends a backup task to the first backup agent, the first backup agent starts the coordination module, the backup agent obtains the file message to be backed up from the distributed message queue of the coordination module, and sends a status update message to the status distributed message queue after the backup is completed.
[0011] Preferably, when the first proxy fails, the main control unit sends a backup task to the second backup proxy and starts the coordination module of the second backup proxy.
[0012] Preferably, when the backup module of the backup agent takes out the file messages to be backed up from the message queue for backing up, it can take out multiple file messages at a time.
[0013] Preferably, when the number of backup proxies is two or more, the maximum backup speed of a single proxy and the maximum read speed of the backup source are calculated to obtain the required number of backup proxies. The maximum read speed of the backup source is obtained using a disk speed measurement program, and the maximum backup speed of the single proxy is obtained by actually performing a backup. The required number of backup proxies = round (maximum read speed of the backup source / maximum speed of the single proxy backup).
[0014] Preferably, the resource monitoring module sets the thread to sleep and wait when acquiring the message in the message queue.
[0015] The present invention further provides a backup method for a load-balanced, massive file multi-agent concurrent backup system, comprising: The main control unit sends the backup task to a backup agent; The coordination module of the backup agent that receives the task scans the massive file directory and notifies all backup agents to start the backup; The backup agent that receives the notification message obtains the file message in the task distributed message queue; Proxy backup files and update status distributed message queues; The resource monitoring module notifies the backup module to adjust the speed of fetching file messages; The backup is complete.
[0016] This load-balanced massive file multi-agent concurrent backup system and method realizes load-balanced backup of multiple agents, avoiding the problem of large differences in the execution time of multiple agent backup jobs; avoiding the problem that the number of agents is limited in special scenarios, and adding or reducing agents will lead to poor allocation of sub-directories to be backed up; when the directory structure of the massive files to be backed up is complex, the massive files to be backed up can be reasonably allocated to multiple agents for concurrent backup, and the backup completion time of each backup agent is basically the same; there is no need to scan before backup, and there will be no problem that the number of backup agents obtained according to the scan results cannot exert the best concurrent performance. At the same time, there is no limit on the number of backup agents, and they can be increased or decreased according to actual conditions to achieve load balancing during backup agent backup. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 Schematic diagram of the framework of the system of the present invention; Figure 2 Schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0019] The present invention provides Figure 1 The system, described in [1], is a load-balanced, multi-agent concurrent backup system for massive files. It includes a master control unit, backup storage, and multiple backup agents. The backup agents consist of a coordination module, a backup module, a recovery module, and a resource monitoring module. The coordination module consists of a directory scanning thread, a task distributed message queue, and a status distributed message queue. Among the multiple backup agents, one agent initiates the coordination module, which scans the massive file directory to be backed up and stores the scanned files in the task distributed message queue. The backup modules of the multiple agents retrieve the files from the message queue for backup. When an agent completes the file backup, it sends a status update message to the status distributed message queue. The agents adjust the speed at which they retrieve messages from the message queue based on their computing resources to avoid excessive resource consumption on their host machines, thus achieving load balancing.
[0020] The main functions of the master control unit are to create backup strategies, issue backup tasks to agents, manage backup sets, manage backup jobs, etc. Backup storage is used to store backup sets; The backup agent is a client installed on the production server that reads the backup data and transfers it to the backup storage; The main function of the coordination module is to quickly and non-duplicatively distribute the massive amount of files to be backed up to the backup modules of each backup agent when multiple agents are backing up concurrently. The backup module transfers the data to be backed up to the backup storage; The recovery module transfers the data to be recovered from the backup storage to the recovery target location; The resource monitoring module monitors the computing resources of the host where the backup agent is located during backup and recovery, adjusts the backup and recovery speed, reduces the business impact on the host where the agent is located, and appropriately adds thread sleep waiting when obtaining messages in the message queue to reduce the speed of obtaining messages.
[0021] Each backup agent includes a coordination module. The master control unit sends a backup task to the first agent, starts the coordination module, and the agent obtains the file message to be backed up from the distributed message queue of the coordination module. After the backup is completed, it sends a status update message to the status distributed message queue. When the first agent fails, the backup strategy is still valid. The main control unit will send the backup task to the second agent, start the coordination module of the second backup agent, and so on.
[0022] When the backup module of the backup agent takes out the file messages to be backed up from the message queue for backup, it can take multiple file messages at a time to reduce network IO consumption; The proxy adjusts the speed of fetching messages from the message queue based on its own computing resources to avoid excessive resource consumption on the host and achieve load balancing. Install multiple backup proxies. Estimate the number of backup proxies required based on the maximum backup speed of a single proxy and the maximum read speed of the backup source. This number can be less than the specified number. The maximum read speed of the backup source can be determined using disk speed measurement software, and the maximum backup speed of a single proxy can be determined by performing a backup. The number of backup proxies = round(maximum read speed of the backup source / maximum backup speed of a single proxy). Set up backup policies on the master control unit, select the massive file directories to be backed up, choose the backup agent to perform the backup, and configure other parameters such as compression, encryption, rate limiting, deduplication, and scheduling; The master control unit sends a backup task to the first backup agent, activating its coordination module. This module scans the massive file directory to be backed up, stores the files found in the task's distributed message queue, and notifies the backup modules of all agents to retrieve the files from the message queue for backup. When an agent completes the file backup, it sends a status update message to the status distributed message queue.
[0023] The backup module of the backup agent uses multi-threaded backup technology to maximize the backup performance of a single agent. The backup agent adjusts the speed of fetching messages from the message queue according to the computing resources monitored by the resource monitoring module to avoid excessive consumption of host resources.
[0024] The present invention further provides Figure 2 The load-balanced multi-agent concurrent backup method for massive files shown includes the following steps: include: The main control unit sends the backup task to a backup agent; The coordination module of the backup agent that receives the task scans the massive file directory and notifies all backup agents to start the backup; The backup agent that receives the notification message obtains the file message in the task distributed message queue; Proxy backup files and update status distributed message queues; The resource monitoring module notifies the backup module to adjust the speed of fetching file messages; The backup is complete.
[0025] This load-balanced massive file multi-agent concurrent backup system and method realizes load-balanced backup of multiple agents, avoiding the problem of large differences in the execution time of multiple agent backup jobs; avoiding the problem that the number of agents is limited in special scenarios, and adding or reducing agents will lead to poor allocation of sub-directories to be backed up; when the directory structure of the massive files to be backed up is complex, the massive files to be backed up can be reasonably allocated to multiple agents for concurrent backup, and the backup completion time of each backup agent is basically the same; there is no need to scan before backup, and there will be no problem that the number of backup agents obtained according to the scan results cannot exert the best concurrent performance. At the same time, there is no limit on the number of backup agents, and they can be increased or decreased according to actual conditions to achieve load balancing during backup agent backup.
[0026] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A load-balanced, multi-agent concurrent backup system for massive files, characterized by: include: The main control unit is used to create backup strategies, send backup tasks, manage backup sets, and manage backup jobs; Backup storage, used to store the backup set; at least one backup agent, configured to receive and execute backup tasks from the master control unit and transfer data to the backup storage; The backup agent includes: A backup module, used to transfer the data to be backed up to the backup storage; A coordination module, configured to quickly and non-duplicatively distribute the massive amount of files that need to be backed up to the backup module of each backup agent when multiple agents perform concurrent backups; A recovery module, configured to transfer the data to be recovered from the backup storage to a recovery target location; A resource monitoring module is used to monitor the computing resources of the host where the backup agent is located and adjust the backup and recovery speed during backup and recovery; Among them, the client is installed on the production server.
2. The load-balanced, multi-agent concurrent backup system for massive files according to claim 1, characterized in that: The coordination module includes: Directory scanning thread, task distributed message queue, and status distributed message queue.
3. The load-balanced, multi-agent concurrent backup system for massive files according to claim 2, characterized in that: The directory scanning thread scans the massive file directories to be backed up, and stores the scanned files to be backed up in the task distributed message queue. The backup modules of multiple backup agents take out the file messages to be backed up from the message queue for backing up. When the backup agent completes backing up the files, it sends a status update message to the status distributed message queue.
4. The load-balanced, multi-agent concurrent backup system for massive files according to claim 3, characterized in that: When the number of the backup agents is two or more, one backup agent is selected to start the coordination module.
5. The load-balanced, multi-agent concurrent backup system for massive files according to any one of claims 1 to 4, characterized in that: The main control unit sends a backup task to the first backup agent, the first backup agent starts the coordination module, and other backup agents obtain file messages to be backed up from the distributed message queue of the coordination module, and send status update messages to the status distributed message queue after the backup is completed.
6. The load-balanced, multi-agent concurrent backup system for massive files according to claim 5, characterized in that: When the first backup agent fails, the main control unit sends a backup task to the second backup agent and starts the coordination module of the second backup agent.
7. The load-balanced, multi-agent concurrent backup system for massive files according to claim 6, characterized in that: When the backup module of the backup agent takes out the file messages to be backed up from the message queue for backing up, multiple file messages may be taken at one time.
8. The load-balanced, multi-agent concurrent backup system for massive files according to claim 1, characterized in that: When there are two or more backup proxies, the required number of backup proxies is obtained by calculating the maximum backup speed of a single proxy and the maximum read speed of the backup source. The maximum read speed of the backup source is obtained using a disk speed measurement program. The maximum backup speed of a single proxy is obtained by actually performing a backup. The required number of backup proxies = round(maximum read speed of the backup source / maximum backup speed of a single proxy).
9. The load-balanced, multi-agent concurrent backup system for massive files according to claim 1, characterized in that: When the resource monitoring module obtains the message in the message queue, it sets the thread to sleep and wait.
10. The backup method of the load-balanced massive file multi-agent concurrent backup system according to any one of claims 1 to 9, characterized in that: include: The main control unit sends the backup task to a backup agent; The coordination module of the backup agent that receives the task scans the massive file directory and notifies all backup agents to start the backup; The backup agent that receives the notification message obtains the file message in the task distributed message queue; Proxy backup files and update status distributed message queues; The resource monitoring module notifies the backup module to adjust the speed of fetching file messages; The backup is complete.