Distributed file-level backup method and system based on object storage

Abstract

The present invention discloses a distributed file-level backup method and system based on object storage, which belongs to the technical field of object storage and container mirroring. The technical problem to be solved by the present invention is that as the amount of data increases, the backup time will be long, the backup data will be repeated, the synchronization of multiple copies will be slow, and the backup storage cost will be high. The technical solution adopted is that the method cuts the data into blocks, saves the block information to the task queue for file-level backup, and provides all backup servers for consumption; before each backup, the data blocks will be compared, and when a database with the same MD5 has been backed up, the current database will not be backed up, ensuring that only one identical data will be saved; the details are as follows: generating a system image; system initialization; backup task initialization; data backup; monitoring system; backup and recovery.

Description

Technical Field

[0001] The present invention relates to the technical field of object storage and container mirroring, and in particular to a distributed file-level backup method and system based on object storage. Background Art

[0002] Object storage, also known as object-based storage, is a general term used to describe an approach to addressing and processing discrete units called objects. Like files, objects contain data, but unlike files, objects are not hierarchical in a hierarchy. Each object is at the same level in a flat address space called a storage pool, and one object does not belong to another object below it. Both files and objects have metadata associated with the data they contain, but objects are characterized by extended metadata. Each object is assigned a unique identifier, allowing a server or end user to retrieve the object without having to know the physical location of the data. This approach helps automate and simplify data storage in cloud computing environments.

[0003] File backup refers to the process of copying all or part of a file collection from the application host's hard disk or array to other storage media to prevent file loss due to system operation errors or system failures.

[0004] With the development of the Internet and big data, the amount of data storage has increased exponentially. More and more data needs to be stored, and file backup has become an indispensable requirement for data storage, especially in Internet projects. Existing backup methods include disk backup, virtual machine backup, multiple copies, etc. They have the advantages of fast response and high availability. However, as the amount of data increases, problems such as long backup time, duplicate backup data, slow synchronization of multiple copies, and high backup storage costs become increasingly prominent. Summary of the Invention

[0005] The technical task of the present invention is to provide a distributed file-level backup method and system based on object storage to solve the problems of long backup time, duplicate backup data, slow synchronization of multiple copies, and high backup storage costs as the amount of data increases.

[0006] The technical task of the present invention is achieved in the following way: a distributed file-level backup method based on object storage, which performs file-level backup by slicing data and saving the block information to a task queue (RabbitMQ queue), and provides all backup servers for consumption; before each backup, a data block comparison is performed. If a database with the same MD5 has been backed up, the current database will not be backed up, ensuring that only one identical data is saved; the details are as follows:

[0007] Generate system image;

[0008] System initialization;

[0009] Backup task initialization;

[0010] Data backup;

[0011] Monitoring system;

[0012] Backup and recovery.

[0013] As a preference, the system image is generated as follows:

[0014] The server-side backup system and monitoring system are generated into a server-side image A containing the server-side backup system and monitoring system by using the container engine Docker;

[0015] At the same time, the container engine docker is used to generate image B containing the client application.

[0016] As a preference, the system initialization is as follows:

[0017] According to specific needs, write the number of server backup systems required by the current environment into the configuration file;

[0018] Use container technology and server image A to create a specified number of server backup systems.

[0019] Use the form of intranet virtual IP (VIP) to fix the IP address of the server-side server to ensure normal network communication between each server;

[0020] The client uses mirror B to create a server containing the client according to user requirements, and directly uses the method of installing the client application to connect the client and server networks to ensure smooth two-way network communication.

[0021] Preferably, the backup task initialization includes obtaining files and issuing tasks; specifically, the following steps:

[0022] The backup task will be sent to any backup server randomly;

[0023] After receiving the instruction, the backup server (server) parses the corresponding client address in the instruction and sends the backup instruction to the corresponding client application (agent);

[0024] The client application (agent) receives the backup request, parses the file, file list, or folder to be backed up in the request, calculates the MD5 value of each file in 4MB chunks (the size can be set based on the scenario), and returns all the calculation results to the auxiliary backup system.

[0025] The backup system saves the correspondence between files and blocks, the MD5 values corresponding to the blocks, and the metadata information of the files into a metadata file, uploads it to the file in the object storage, and stores it in a directory with the task id as the folder;

[0026] The backup server will publish all block information (including MD5 values, files corresponding to data blocks, and locations) to the RabbitMQ message queue for consumption by the backup system. After each backup is completed, the backup system will return the upload completion information so that the upper layer can determine whether the backup task of any task ID is completed.

[0027] As a preference, data backup is as follows:

[0028] When all backup systems detect that there are messages available for consumption in the message queue, they will obtain a group of messages (5 messages, the specific size can be adjusted according to the number) from the queue for execution;

[0029] Temporarily save the relevant information of the group's consumption data to the object storage. The file will be saved in a file named after the backup server's virtual IP (VIP) in the tmp folder. After the temporary file is uploaded, the actual data upload task is executed using an asynchronous thread to perform the backup, ensuring that the message is consumed quickly.

[0030] Parse the received message and determine whether the md5 value in the data set exists by using the object storage method to determine whether the object exists (Note: in object storage, data blocks are named by their md5 values):

[0031] If the corresponding object exists in the object storage, the upload task of this data block is skipped and the next block is uploaded;

[0032] When a group of messages is consumed, first clean up the temporary files in the tmp folder, then go to the message queue to get a group of data, and repeat the upload process until the backup is completed (all messages in the message queue are consumed).

[0033] As a preference, the monitoring system is specifically as follows:

[0034] Each backup server has a data backup system and a monitoring system. The monitoring system is divided into a primary monitoring system and a secondary monitoring system. The primary monitoring system regularly checks the availability of all backup servers, the availability of the backup systems, the status of the secondary monitoring systems, and the status of the primary monitoring system. The monitoring system has two monitoring paths: one is the primary monitoring system monitoring the backup servers, backup systems, and secondary monitoring systems; the other is the secondary monitoring system monitoring the primary monitoring system. The monitoring process is as follows:

[0035] After the entire backup system is deployed, the main monitoring task will perform status checks regularly. Specifically, the main monitoring system will check whether the backup server is normal. If an exception occurs, the main monitoring system will first obtain the backup task files currently being executed by the abnormal backup server from the object storage, assemble the tasks, and republish them to the message queue (skip publishing if there are no files), and delete the files from the object storage so that other normal backup systems can consume them. After publishing, the main monitoring system will delete the abnormal backup server and use the created image to generate a server with the same virtual IP after all detection tasks (including backup server detection, backup system detection, and auxiliary monitoring system detection of the entire main monitoring system) are completed.

[0036] After the server check is complete, the main monitoring system will perform an availability check on the backup system by calling the check interface exposed by the backup system to perform status detection: If an abnormal backup system is found, the main monitoring system will restart the abnormal backup system and record the restart count (if the next check is normal, the restart count record will be reset to 0, otherwise it will be accumulated by 1). If the restart count exceeds 3 times, the main monitoring system will delete the server where the backup system is located and use the created image to generate a server with the same virtual IP address;

[0037] After the backup system is tested, the primary monitoring system will test the secondary monitoring system. If any secondary monitoring system is detected to be abnormal, the secondary monitoring system will be restarted and the number of restarts will be recorded. As with the backup system, if the number of restarts reaches the threshold, an alarm will be issued.

[0038] While the main monitoring system is checking the backup server and backup system, the auxiliary monitoring system will check the main monitoring system: if the main monitoring system is found to be abnormal, the auxiliary monitoring system will compete for a main monitoring system and take over the tasks of the main monitoring system.

[0039] Preferably, the backup and recovery are as follows:

[0040] The backup system will obtain a file containing all files backed up this time, file metadata information, and MD5 values of file blocks from the object storage based on the task ID;

[0041] Publish all file information that needs to be restored to the message queue for data recovery according to the file granularity;

[0042] The backup system obtains messages from the message queue, parses the data to be restored in the message, obtains files corresponding to MD5 values from the object storage, stitches them together into a complete file in sequence, and restores the metadata information;

[0043] The file temporarily storing the data is deleted from the object storage, and the recovery completion information is returned to the upper-layer application for statistical purposes.

[0044] A distributed file-level backup system based on object storage, which includes a task queue (RabbitMQ queue), a client application (agent) and multiple backup servers. The backup servers include a monitoring system (monitor) and a backup system (server). The monitoring systems communicate with each other and randomly select one as the main monitoring system. The main monitoring system is used to monitor the backup server and monitor and create the auxiliary monitoring system. The auxiliary monitoring system monitors the main monitoring system. When the main monitoring system is found to be unavailable, one is randomly selected from the remaining monitoring systems as the main monitoring system; when the backup server goes down, the main monitoring system will clean up the down backup server and regenerate a new backup server, thereby ensuring high availability and high reliability of the backup.

[0045] Preferably, the monitoring system will perform an inspection task at regular intervals. The monitoring system inspection steps include checking the server status, checking the backup system status, checking the auxiliary monitoring system status, and checking the main monitoring system status, as follows:

[0046] The main monitoring system first checks whether all backup servers are operating normally according to the VIP address list configured in the configuration file:

[0047] If any server is detected to have network disconnection, the main monitoring system will first delete the problematic server and use the generated server image to create a VIP-like backup server;

[0048] When the server is detected to be faulty, the backup system inside the server is checked, and the backup system exposes a health check interface for the main monitoring system to check. If a problem is found in the backup system, the main monitoring system will obtain an object named after the server's virtual IP (VIP) from the object storage, republish the object's content to the message queue (RabbitMQ), and then attempt to restart the problematic backup system. If the restart fails, the main monitoring system will retry (up to 3 times). If the number of retries reaches the threshold and the backup system still fails to start normally, the main monitoring system will clean up the server where the backup system is located, generate a new backup server, and re-execute the backup system detection task.

[0049] After completing the detection of the backup server and backup system, the main monitoring system will check the status of the auxiliary monitoring system. When the auxiliary monitoring system is found to be in an abnormal state, the main monitoring system will restart it. If the auxiliary monitoring system has not started normally when the number of restarts reaches the threshold, an alarm will be issued. At the same time, the auxiliary monitoring system will monitor the main monitoring system regularly. When the main monitoring system is found to be unavailable, one of the auxiliary monitoring systems will be selected as the main monitoring system through competition and will take over the work of the main monitoring system.

[0050] More preferably, the client application (agent) calculates the MD5 value of each block of the locally backed-up data in 4MB chunks, and sends the MD5 value and file correspondence (the MD5 sequence of the file's MD5 and the file content sequence) to the backup system. The backup system uses the received information to create a file in the object store containing the backup file and the MD5 correspondence, using the backup id, and publishes it to the message queue (RabbitMQ). All backup systems consume the message from the queue, and the backup system queries the object store based on the MD5 value to see if a corresponding object exists (i.e., the object name is named after the MD5 value. If an object with the same name as the MD5 value appears, it can be determined that the data block already exists):

[0051] If it does not exist, it will be uploaded using the md5 value name;

[0052] Otherwise, the upload is skipped, thus ensuring the uniqueness of the data blocks stored in the object storage and reducing storage space and upload traffic.

[0053] The object-based distributed file-level backup method and system of the present invention have the following advantages:

[0054] (1) The present invention adopts a systematic backup method, which can reduce the energy required by users to upload files. It adopts a distributed data processing method to reduce the time required for users to back up and restore data. It adopts the form of data block backup to perform data deduplication, reducing the traffic and space usage during the user backup process, thereby effectively improving the user experience. At the same time, for storage devices, due to the high availability and low redundancy of data, the number of disk erases will be greatly reduced, thereby increasing the service life of the disk;

[0055] (2) The backup data of the present invention is divided into blocks, and the block information is saved in the RabbitMQ queue, which is provided to all backup servers for consumption. This avoids the situation where the backup time of a server is too long and other servers are idle due to uneven file sizes. The use of the RabbitMQ queue can also avoid task interruptions and the need to restart the entire task. By calculating the MD5 of the file block, each data block will be compared before uploading, reducing request traffic and storage space usage, and improving data utilization.

[0056] (3) The present invention fully utilizes the high reliability, high performance, and high concurrency characteristics of object storage, and uses distributed servers to provide a highly concurrent backup method. File-level backup has the characteristics of high backup flexibility, a relatively small number of backup files, and a fast backup speed. At the same time, object storage, as a distributed storage, has the advantages of large capacity, high security, and low cost.

[0057] (IV) The present invention effectively increases the fault tolerance during the user's file backup process, effectively reduces the time required for users to back up file data by adopting a distributed multi-server backup method; and effectively avoids duplicate backup of duplicate data during the backup process by using block calculation MD5 technology.

[0058] It has the characteristics of reasonable design, simple structure, easy processing, small size, easy use, and multiple uses. Therefore, it has great promotion and use value. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention will be further described below with reference to the accompanying drawings.

[0060] Attachment Figure 1 This is a schematic diagram of the relationship between the various parts of the distributed file-level backup system based on object storage;

[0061] Attachment Figure 2 A flowchart for initializing a backup task;

[0062] Attachment Figure 3 A flowchart for data backup;

[0063] Attachment Figure 4 It is the flow chart of the monitoring system;

[0064] Attachment Figure 5 A flowchart of the backup and recovery process. DETAILED DESCRIPTION

[0065] The following is a detailed description of a distributed file-level backup method and system based on object storage according to the present invention with reference to the accompanying drawings and specific embodiments.

[0066] Example 1

[0067] The object-based distributed file-level backup method of the present invention performs file-level backup by slicing data and saving the block information to a task queue (RabbitMQ queue), and provides consumption to all backup servers. Before each backup, data blocks are compared. If a database with the same MD5 has been backed up, the current database will not be backed up, ensuring that only one identical data is saved. The details are as follows:

[0068] S1. Generate system image;

[0069] S2, system initialization;

[0070] S3, backup task initialization;

[0071] S4, data backup;

[0072] S5, monitoring system;

[0073] S6. Backup and recovery.

[0074] In this embodiment, the generation of the system image in step S1 is specifically as follows:

[0075] S101, using the container engine Docker to generate a server-side image A containing the server-side backup system and the monitoring system;

[0076] S102: Generate a mirror image B containing a client application using the container engine Docker.

[0077] In this embodiment, the system initialization of step S2 is specifically as follows:

[0078] S201. Write the number of server backup systems required by the current environment into the configuration file according to specific needs;

[0079] S202: Using container technology, create a specified number of server backup systems using server image A.

[0080] S203, using the form of intranet virtual IP (VIP) to fix the IP address of the server-side server to ensure normal network communication between each server;

[0081] S204: The client uses image B to create a server containing the client according to the user's requirements, and directly uses the client application to connect the client and server networks to ensure smooth two-way network communication.

[0082] As attached Figure 2 As shown, the backup task initialization in step S3 of this embodiment includes obtaining files and issuing tasks; the details are as follows:

[0083] ①. Send the address of the machine to be backed up and the files to be backed up: the backup task will be randomly sent to any backup server;

[0084] ② Send the files to be backed up to the agent: After receiving the instruction, the backup server (server) parses the corresponding client address in the instruction and sends the backup instruction to the corresponding client application (agent);

[0085] ③. Obtain the MD5 value of the file to be backed up and the corresponding relationship between the files in 4MB chunks, and send it to the backup system;

[0086] ④. Upload the obtained file, the corresponding relationship between the MD5 value, and the file metadata information to the object storage file, and use the backup ID as the directory where the file is located;

[0087] ⑤. Assemble the obtained files and the corresponding MD5 values and send them to the message queue.

[0088] As attached Figure 3 As shown, the data backup in step S4 of this embodiment is specifically as follows:

[0089] ①. Get a group of messages (5 messages in a group) from the backup information queue;

[0090] ②. Temporarily save the message parsed in step ① to the object storage tmp folder for use in backing up the system or republishing the message when the server fails.

[0091] ③. Parse the message obtained in step ① and use the parsed MD5 value to determine whether there is an object named with this MD5 value in the object storage:

[0092] If it exists, skip uploading;

[0093] ④. Call the agent to obtain the data of the corresponding database;

[0094] ⑤. Upload the data information to the object storage, using the MD5 value as the object name. After the upload is complete, repeat step ② until the acquired message is consumed, then proceed to step ⑥.

[0095] ⑥. Delete the file temporarily storing the data from the object storage, and return the uploaded database information to the upper-level application for statistics.

[0096] As attached Figure 4 As shown, the monitoring system of step S5 in this embodiment is specifically as follows:

[0097] Each backup server has a data backup system and a monitoring system. The monitoring system is divided into a primary monitoring system and a secondary monitoring system. The primary monitoring system regularly checks the availability of all backup servers, the availability of the backup systems, the status of the secondary monitoring systems, and the status of the primary monitoring system. The monitoring system has two monitoring paths: one is the primary monitoring system monitoring the backup servers, backup systems, and secondary monitoring systems; the other is the secondary monitoring system monitoring the primary monitoring system. The monitoring process is as follows:

[0098] After the entire backup system is deployed, the main monitoring task will perform status checks regularly. Specifically, the main monitoring system will check whether the backup server is normal. If an exception occurs, the main monitoring system will first obtain the backup task files currently being executed by the abnormal backup server from the object storage, assemble the tasks, and republish them to the message queue (skip publishing if there are no files), and delete the files from the object storage so that other normal backup systems can consume them. After publishing, the main monitoring system will delete the abnormal backup server and use the created image to generate a server with the same virtual IP after all detection tasks (including backup server detection, backup system detection, and auxiliary monitoring system detection of the entire main monitoring system) are completed.

[0099] After the server check is complete, the main monitoring system will perform an availability check on the backup system by calling the check interface exposed by the backup system to perform status detection: If an abnormal backup system is found, the main monitoring system will restart the abnormal backup system and record the restart count (if the next check is normal, the restart count record will be reset to 0, otherwise it will be accumulated by 1). If the restart count exceeds 3 times, the main monitoring system will delete the server where the backup system is located and use the created image to generate a server with the same virtual IP address;

[0100] After the backup system is tested, the primary monitoring system will test the secondary monitoring system. If any secondary monitoring system is detected to be abnormal, the secondary monitoring system will be restarted and the number of restarts will be recorded. As with the backup system, if the number of restarts reaches the threshold, an alarm will be issued.

[0101] While the main monitoring system is checking the backup server and backup system, the auxiliary monitoring system will check the main monitoring system: if the main monitoring system is found to be abnormal, the auxiliary monitoring system will compete for a main monitoring system and take over the tasks of the main monitoring system.

[0102] As attached Figure 5 As shown, the backup and recovery of step S6 in this embodiment is specifically as follows:

[0103] ①. Get an appropriate number of messages from the recovery message queue;

[0104] ②. Temporarily save the message parsed in step ① to the object storage tmp folder for use in backing up the system or republishing the message when the server fails.

[0105] ③. Obtain data blocks from the object storage according to the MD5 sequence in the message and send the data to the agent for storage. After all data is recovered, restore the file's data information (such as permissions).

[0106] ④. Delete the file that temporarily stores data from the object storage, and return the recovery completion information to the upper-level application for statistics.

[0107] The backup execution strategy can be scheduled single execution or scheduled periodic execution, and only one strategy can exist at a time.

[0108] Example 2:

[0109] As attached Figure 1 As shown, the distributed file-level backup system based on object storage of the present invention includes a task queue (RabbitMQ queue), a client application (agent) and multiple backup servers. The backup server includes a monitoring system (monitor) and a backup system (server). The monitoring systems communicate with each other and randomly select one as the main monitoring system. The main monitoring system is used for backup server monitoring, auxiliary monitoring system monitoring and creation, and the auxiliary monitoring system monitors the main monitoring system. When the main monitoring system is found to be unavailable, one is randomly selected from the remaining monitoring systems as the main monitoring system; when the backup server goes down, the main monitoring system will clean up the down backup server and regenerate a new backup server, thereby ensuring high availability and high reliability of the backup.

[0110] The monitoring system in this embodiment will perform a check task at regular intervals. The monitoring system check steps include checking the server status, checking the backup system status, checking the auxiliary monitoring system status, and checking the main monitoring system status. The details are as follows:

[0111] The main monitoring system first checks whether all backup servers are operating normally according to the VIP address list configured in the configuration file:

[0112] If any server is detected to have network disconnection, the main monitoring system will first delete the problematic server and use the generated server image to create a VIP-like backup server;

[0113] When the server is detected to be faulty, the backup system inside the server is checked, and the backup system exposes a health check interface for the main monitoring system to check. If a problem is found in the backup system, the main monitoring system will obtain an object named after the server's virtual IP (VIP) from the object storage, republish the object's content to the message queue (RabbitMQ), and then attempt to restart the problematic backup system. If the restart fails, the main monitoring system will retry (up to 3 times). If the number of retries reaches the threshold and the backup system still fails to start normally, the main monitoring system will clean up the server where the backup system is located, generate a new backup server, and re-execute the backup system detection task.

[0114] After completing the detection of the backup server and backup system, the main monitoring system will check the status of the auxiliary monitoring system. When the auxiliary monitoring system is found to be in an abnormal state, the main monitoring system will restart it. If the auxiliary monitoring system has not started normally when the number of restarts reaches the threshold, an alarm will be issued. At the same time, the auxiliary monitoring system will monitor the main monitoring system regularly. When the main monitoring system is found to be unavailable, one of the auxiliary monitoring systems will be selected as the main monitoring system through competition and will take over the work of the main monitoring system.

[0115] In this embodiment, the client application (agent) calculates the MD5 value of each block of the locally backed-up data in 4MB chunks, and sends the MD5 value and file correspondence (the MD5 sequence of the file contents corresponding to the MD5 sequence) to the backup system. The backup system uses the received information to create a file in the object store containing the backup file and the MD5 correspondence, using the backup id, and publishes it to the message queue (RabbitMQ). All backup systems consume the message from the queue, and the backup system queries the object store based on the MD5 value to see if a corresponding object exists (that is, the object name is named after the MD5 value. If an object with the same name as the MD5 value appears, it can be determined that the data block already exists):

[0116] If it does not exist, it will be uploaded using the md5 value name;

[0117] Otherwise, the upload is skipped, thus ensuring the uniqueness of the data blocks stored in the object storage and reducing storage space and upload traffic.

[0118] Among them, the principle of this backup system is: use object storage as the storage space for file backup, make full use of the high reliability, high performance, high concurrency, fast reading and writing of object storage, and realize the efficiency and reliability of file backup. It uses scheduled tasks to perform scheduled file backup and uses multi-server backup to realize distributed backup, thereby greatly improving the user experience.

[0119] The object-based file-level backup system of the present invention includes the following components and dependencies: an application-internal plug-in, a system controller, an object storage SDK, a message queue, a server, a virtual IP, a Docker container engine, and an object storage service system. The file upload and download SDK includes, but is not limited to, file upload, file download, and file metadata retrieval; the message queue includes, but is not limited to, RabbitMQ; the backup system includes, but is not limited to, full and incremental backups; and backup strategies include, but are not limited to, manual and scheduled execution.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed file-level backup method based on object storage, characterized in that: This method is to cut the data into blocks, save the block information to the task queue for file-level backup, and provide it to all backup servers for consumption. Before each backup, the data blocks will be compared. If a database with the same MD5 has been backed up, the current database will not be backed up, ensuring that only one identical data is saved. The details are as follows: Generate system image; System initialization; Backup task initialization; Data backup; Monitoring system; Backup and recovery; The monitoring system is as follows: Each backup server has a data backup system and a monitoring system. The monitoring system is divided into a primary monitoring system and a secondary monitoring system. The primary monitoring system regularly checks the availability of all backup servers, the availability of the backup system, and the status of the secondary monitoring system. The monitoring system has two monitoring paths: one is the primary monitoring system monitoring the backup servers, backup systems, and secondary monitoring systems; the other is the secondary monitoring system monitoring the primary monitoring system. The monitoring process is as follows: After the entire backup system is deployed, the main monitoring task will perform regular status checks. Specifically, the main monitoring system will check whether the backup server is normal. If an exception occurs, the main monitoring system will first obtain the backup task files currently being executed by the abnormal backup server from the object storage, assemble the tasks, and republish them to the message queue. The files will then be deleted from the object storage so that other normal backup systems can consume them. After the release is complete, the main monitoring system will delete the abnormal backup server and, after all detection tasks are completed, use the created image to generate a server with the same virtual IP. After the server check is complete, the main monitoring system will perform an availability check on the backup system by calling the check interface exposed by the backup system to perform status detection. If an abnormal backup system is found, the main monitoring system will restart the abnormal backup system and record the restart count. If the restart count exceeds three times, the main monitoring system will delete the server where the backup system is located and use the created image to generate a server with the same virtual IP address. After the backup system is tested, the primary monitoring system will test the secondary monitoring system. If any secondary monitoring system is detected to be abnormal, the secondary monitoring system will be restarted and the number of restarts will be recorded. As with the backup system, if the number of restarts reaches the threshold, an alarm will be issued. While the primary monitoring system is checking the backup server and backup system, the secondary monitoring system will also check the primary monitoring system. If an abnormality is detected in the primary monitoring system, the secondary monitoring system will compete for a primary monitoring system and take over the primary monitoring system's tasks. Generate a system image as follows: The server-side backup system and monitoring system are generated into a server-side image A containing the server-side backup system and monitoring system by using the container engine Docker; At the same time, the container engine Docker is used to generate the image B containing the client application; The system initialization is as follows: Write the number of server backup systems required by the current environment into the configuration file; Use container technology and server image A to create a specified number of server backup systems. Use the form of intranet virtual IP to fix the IP address of the server; The client uses image B to create a server containing the client according to user requirements, and directly installs the client application to connect the client and server networks. Backup task initialization includes obtaining files and issuing tasks; the details are as follows: The backup task will be sent to any backup server randomly; After receiving the instruction, the backup server parses the corresponding client address in the instruction and sends the backup instruction to the corresponding client application; The client application receives the backup request, parses the file, file list, or folder to be backed up in the request, calculates the MD5 value of each 4MB file, and returns all the calculation results to the secondary backup system. The backup system saves the correspondence between files and blocks, the MD5 values corresponding to the blocks, and the metadata information of the files into a metadata file, uploads it to the file in the object storage, and stores it in a directory with the task id as the folder; The backup server will publish all block information to the RabbitMQ message queue for consumption by the backup system. After each backup is completed, the backup system will return the upload completion information so that the upper layer can determine whether the backup task of any task ID is completed; The data backup is as follows: When all backup systems detect that there are messages available for consumption in the message queue, they will obtain a group of messages from the queue for execution; Temporarily save the relevant information of the corresponding group's consumption data to the object storage. The file will be saved in the tmp folder and named after the virtual IP of the backup server. After the temporary file is uploaded, the actual data upload task is executed using an asynchronous thread to perform the backup and ensure that the message is consumed quickly. Parse the received message and determine whether the object exists by using the object storage method to determine whether the md5 value in the corresponding group data exists: If the corresponding object exists in the object storage, the upload task of the corresponding data block is skipped and the next block is uploaded; When a group of messages is consumed, first clean up the temporary files in the tmp folder, then go to the message queue to get a group of data, and repeat the upload process until the backup is completed; the backup and recovery are as follows: The backup system will obtain a file containing all files backed up this time, file metadata information, and MD5 values of file blocks from the object storage based on the task ID; Publish all file information that needs to be restored to the message queue for data recovery according to the file granularity; The backup system obtains messages from the message queue, parses the data to be restored in the message, obtains files corresponding to MD5 values from the object storage, stitches them together into a complete file in sequence, and restores the metadata information; The file temporarily storing the data is deleted from the object storage, and the recovery completion information is returned to the upper-layer application for statistical purposes.

2. A distributed file-level backup system based on object storage, characterized in that: The system is used to implement the object storage-based distributed file-level backup method according to claim 1; the system includes a task queue, a client application, and multiple backup servers, the backup servers including a monitoring system and a backup system, the monitoring systems communicate with each other and randomly select one as a primary monitoring system, the primary monitoring system is used to monitor the backup server, and the secondary monitoring system is used to monitor and create the secondary monitoring system, the secondary monitoring system monitors the primary monitoring system, and when the primary monitoring system is found to be unavailable, a random selection is made from the remaining monitoring systems to serve as the primary monitoring system; When the backup server goes down, the main monitoring system will clean up the down backup server and regenerate a new backup server, thus ensuring high availability and high reliability of the backup; The monitoring system will perform an inspection task at regular intervals. The inspection steps include checking the server status, checking the backup system status, checking the auxiliary monitoring system status, and checking the main monitoring system status. The details are as follows: The main monitoring system first checks whether all backup servers are operating normally according to the VIP address list configured in the configuration file: If any server is detected to have network disconnection, the main monitoring system will first delete the problematic server and use the generated server image to create a VIP-like backup server; When the server is detected to be faulty, the backup system inside the server is checked, and the backup system exposes a health check interface for the main monitoring system to check. If a problem is found in the backup system, the main monitoring system will obtain an object named after the virtual IP address of this server from the object storage, republish the content of the object to the message queue, and then try to restart the problematic backup system. If the restart fails, the main monitoring system will retry. If the number of retries reaches the threshold and the backup system still fails to start normally, the main monitoring system will clean up the server where the backup system is located, generate a new backup server, and re-execute the backup system detection task. After detecting the backup server and backup system, the primary monitoring system will check the status of the secondary monitoring system. If the secondary monitoring system is found to be abnormal, the primary monitoring system will restart it. If the secondary monitoring system fails to start normally when the restart count reaches the threshold, an alarm will be issued. At the same time, the secondary monitoring system will regularly monitor the primary monitoring system. If the primary monitoring system is found to be unavailable, it will select one of the secondary monitoring systems as the primary monitoring system through competition and take over the work of the primary monitoring system. The client application calculates the MD5 value of each 4MB block of local data to be backed up, and sends the MD5 value and file correspondence to the backup system. The backup system uses the received information to create a file in the object storage containing the backup file and the MD5 correspondence, and publishes it to the message queue. All backup systems consume from the message queue. The backup system queries the object storage based on the MD5 value to see if there is a corresponding object: If it does not exist, it will be uploaded using the md5 value name; Otherwise, skip uploading.

Images