File backup method and device based on object storage

Abstract

The invention discloses a file backup method and device based on object storage, and the method comprises the following steps: S1, scanning a source end catalogue, scanning files in the catalogue needing to be backed up, adding the files needing to be backed up into a to-be-transmitted file list of the corresponding catalogue, and storing the files into a database; s2, acquiring the list of files to be transmitted from the database, sending a comparison request to the standby end object storage for each path and the corresponding list of files to be transmitted, acquiring part of files of the standby end according to the source end file name for comparison, and storing the files of the standby end according to the source end file name; adding the modified files of the source end found in the comparison process and the files which do not exist in the standby end into a to-be-uploaded file list, and storing the files into a database according to a source end path for file transmission in a transmission stage; and S3, according to the to-be-uploaded file list, uploading files needing to be backed up to a standby end object for storage.

Description

Technical Field

[0001] The present invention relates to the technical field of computer disaster recovery and backup, and in particular to a file backup method and device based on object storage. Background Art

[0002] With the development of network and computer technology, massive files have entered people's lives. However, there are more and more signs that while using computers to improve production efficiency, the problem of data failure is becoming more and more serious. Once data fails, the company will be in trouble: customer information, financial accounts, technical documents and other data may be destroyed beyond recognition, and the time allowed for recovery is extremely short. If it cannot be recovered, the loss will be heavy. The higher the degree of enterprise informatization, the more important the data backup and recovery measures are, so file backup has become a hot topic.

[0003] Nowadays, more and more companies use computer systems to handle daily business to ease the pressure brought by the increasingly fierce market competition and growing business needs. Computers often generate files according to certain rules. Among the massive files generated every day, there are many files in directories that need to be backed up, and there are also many files in directories that do not need to be backed up. And these directories are interspersed in the same parent directory. In addition, some types of files do not need to be backed up, while some types of files need to be backed up. These files are interspersed in the same directory. Some files remain unchanged after they are generated, while some files are constantly changing. All of this requires some flexible data backup technology.

[0004] At present, object storage is a relatively common object storage method. Object storage is an object-based storage device with intelligent and self-management capabilities. It implements object reading and writing and access to storage resources through web service protocols (such as REST and SOAP). Object storage is an object / file-oriented, massive Internet storage that stores encapsulated files. Therefore, files cannot be opened in the object storage system, but files can be uploaded and downloaded like FTP. In addition, object storage does not have a multi-level file structure like a file system, but only a concept of a bucket. There are many objects stored in the bucket, and each object has its own unique key, which means that knowing the bucket name and the unique key can identify an object. The advantages of object storage are excellent scalability, policy-based automated management features, and support for multi-user management technology.

[0005] Traditional file backup systems back up files in three main stages: scanning, comparison, and transfer. The scanning speed on the source side is relatively fast, but the scanning speed on the backup side is relatively slow, especially when there are many files in the backup side directory, the scanning stage will take too much unnecessary time, and adding a large number of file lists to the memory during the comparison stage often results in insufficient memory. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a file backup method and device based on object storage, so as to realize the scheduled backup of massive files, save memory when there are too many files on the backup side, and greatly reduce the time consumed in scanning the backup side. In the comparison stage, the comparison attributes of the backup side files and the source side files can be determined according to the user's own needs. If the machine performance permits, the number of working threads can be increased to improve the speed of file backup.

[0007] To achieve the above object, the present invention proposes a file backup method based on object storage, comprising the following steps: Step S1, scanning the source directory, scanning the files in the directory to be backed up, adding the files to be backed up to the list of files to be transferred in the corresponding directory and storing them in the database; Step S2, obtaining the list of files to be transmitted from the database, sending a comparison request to the standby object storage for each path and its corresponding list of files to be transmitted, obtaining some files on the standby side for comparison according to the file names on the source side, adding the files that have been modified on the source side and the files that are not on the standby side found in the comparison process to the list of files to be uploaded, and storing them in the database according to the source path for file transmission in the transmission stage; Step S3: uploading the files to be backed up to the backup object storage according to the list of files to be uploaded.

[0008] Preferably, step S1 further comprises: Step S100, scanning the source directory according to the list of directories to be scanned, determining whether the scanned directory needs to be backed up, and if it needs to be backed up, further determining whether it has subdirectories or files; Step S101, if there is a file in the current directory, determine whether the file needs to be backed up. If it needs to be backed up, record the relevant attributes of the file and add it to the list of files to be transferred corresponding to the current directory; Step S102, if there is a subdirectory under the current directory, determine whether the subdirectory meets the scanning condition, and if so, add it to the list of directories to be scanned; Step S103, when the current directory is scanned, the list of files to be transferred in the current directory is stored in the database, and the process returns to step S101 to switch directories for scanning according to the list of directories to be scanned.

[0009] Preferably, each record stored in the database includes a unique identifier (ID), a source end path (path), a json string (str) corresponding to the list of files to be transferred of the source end path, and a standby end path (distpath).

[0010] Preferably, when the number of files scanned in the current directory exceeds a preset threshold, after adding paging information after the source end path, the source end path with the paging information added and the corresponding list of files to be transferred are stored in the database.

[0011] Preferably, step S2 further comprises: Step S200, obtaining a list of files to be transferred from a database, and assigning each source end path and its corresponding list of files to be transferred to different threads for processing; Step S201: for each source end path and its corresponding list of files to be transferred, its corresponding thread sends a comparison request to the backup end object storage to determine whether the backup end path exists. If the backup end path does not exist, all files under the source end path are added to the list of files to be uploaded and saved in the database. Step S202: if the backup end path exists, loop through the list of files to be transmitted of the source end path, obtain files with the same name on the backup end for comparison of specified attributes, and add files with different specified attributes from the files with the same name on the backup end to the list of files to be uploaded.

[0012] Preferably, in step S202, when there is a backup end path, the files in the list of files to be transferred are sequentially checked to obtain the files with the same name on the backup end. If there is no file with the same name on the backup end, the file is added to the list of files to be uploaded. If there is a file with the same name on the backup end, the specified attributes of the two are compared, and the file with different specified attributes from the file with the same name on the backup end is added to the list of files to be uploaded.

[0013] Preferably, the specified attributes include file size, modification timestamp, and md5 value of the file.

[0014] Preferably, in step S202, the method for obtaining the same-name file on the backup end is as follows: Obtain the file with the same name from the backup file list. If there is a file with the same name in the backup file list, take out the corresponding file for comparison, compare the specified attributes of the two, and add the file with different specified attributes from the backup file with the same name to the transfer list; If there is no file with the same name in the backup file list, if the file name is sorted after the last file in the backup file list or the backup file list is empty, a request is sent to the object storage to determine whether there is a file with the same name in the corresponding path on the backup side. If there is a file with the same name, a request is sent to the object storage to obtain a preset threshold number of files after the file using the file as a marker, and the file list returned by the object storage is added to the backup file list, and the file is returned.

[0015] Preferably, in step S3, the list of files to be uploaded of each source end path is obtained from the database, and the lists of files to be uploaded of different directories are allocated to different threads for transmission.

[0016] To achieve the above object, the present invention further provides a file backup device based on object storage, comprising: The scanning module is used to scan the source directory, scan the files in the directory to be backed up, add the files to be backed up to the list of files to be transferred in the corresponding directory and store them in the database; The comparison module obtains the list of files to be transferred from the database, sends a comparison request to the standby object storage for each path and its corresponding list of files to be transferred, obtains some files on the standby side according to the file names on the source side for comparison, adds the files that have been modified on the source side and the files that are not on the standby side found in the comparison process to the list of files to be uploaded, and stores them in the database according to the source path for file transfer in the transmission stage; The upload module uploads the files to be backed up to the backup object storage according to the list of files to be uploaded.

[0017] Compared with the prior art, the object storage-based file backup method and device of the present invention scans the source directory, scans the files in the directory to be backed up, adds the files to be backed up to the list of files to be transferred of the corresponding directory and stores them in the database, then obtains the list of files to be transferred from the database, sends a comparison request to the standby object storage for each source path and its corresponding list of files to be transferred, temporarily obtains the standby files as needed during the comparison process, adds the files that have been modified by the source and the files that are not available on the standby found during the comparison process to the list of files to be uploaded, and stores them in the database according to the source path for file transmission in the transmission stage, and finally uploads the files to be backed up to the object storage of the standby according to the list of files to be uploaded in the transmission stage, thereby achieving the purpose of scheduled backup of massive files. The present invention can save memory and greatly reduce the time consumed by scanning the standby when there are too many files on the standby, and can determine the comparison attributes of the standby files and the source files according to the user's own needs in the comparison stage, and can increase the number of working threads if the machine performance allows, thereby improving the speed of file backup. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a file backup method based on object storage according to the present invention; Figure 2 A system architecture diagram of a file backup device based on object storage according to the present invention; Figure 3 is a flow chart of the scanning phase in an embodiment of the present invention; Figure 4 Flow chart of the comparison phase in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following describes the implementation of the present invention through specific examples and in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0020] Figure 1 FIG. 1 is a flowchart of a file backup method based on object storage according to the present invention. Figure 1 As shown, the present invention provides a file backup method based on object storage, comprising the following steps: Step S1, scanning the source directory, scanning the files in the directory to be backed up, and adding the files to be backed up to the list of files to be transferred in the corresponding directory.

[0021] Specifically, step S1 further includes: Step S100, scanning the source directory according to the list of directories to be scanned, determining whether the scanned directory needs to be backed up, and if it needs to be backed up, further determining whether there are sub-directories or files.

[0022] Specifically, a list of directories to be scanned is established. For example, the root directory is first added to the list of directories to be scanned, and the source directory is scanned according to the list of directories to be scanned. When a directory is scanned, it is determined whether the scanned directory needs to be backed up. If not, the scanning continues. If the currently scanned directory needs to be backed up, it is further determined whether there are subdirectories or files under the current directory.

[0023] Step S101, if there is a file in the current directory, determine whether the file needs to be backed up. If it needs to be backed up, record the file name, file size, timestamp and other related attributes of the file, and add the file to the list of files to be transferred in the current directory.

[0024] Step S102: If there are subdirectories under the current directory, determine whether the subdirectory meets the scanning conditions. If so, add it to the list of directories to be scanned. Because customers sometimes do not want to back up all subdirectories under a directory, some subdirectories may not need to be backed up. Therefore, it is necessary to determine whether each subdirectory meets the scanning conditions (for example, users can set excluded subdirectories), that is, whether it needs to be backed up. If it needs to be backed up, add it to the list of directories to be scanned. Step S103, when the current directory is scanned, the list of files to be transferred in the current directory is stored in the database, and the process returns to step S101 to switch directories for scanning according to the list of directories to be scanned.

[0025] Specifically, assuming that the current directory is d:\abc and the scan is completed, the list of files to be transferred in the current directory records all the files that need to be backed up in the current directory, and the current directory and its list of files to be transferred are stored in the database. The records stored in the database include id (a unique mark of a record), path (source file path, that is, the current directory path), str (the json string corresponding to the list of files to be transferred in the path, and the string can also be compressed), and distpath (the backup path, configured by the user). In a specific embodiment of the present invention, the user only needs to configure the outermost paths of the source and backup ends. After the source is scanned, the subdirectory of the source end can be mapped to the backup end. For example: the source path is d:\abc, the backup end is dtotest: / bcd / , and there are subdirectories b, d, and f under drive d abc. When the list of files to be transferred corresponding to the path d:\abc\b is inserted into the database, its distpath can be automatically mapped to dtotest: / bcd / b.

[0026] Preferably, when the number of files to be backed up scanned in the current directory exceeds a preset threshold (for example, 1000), "#&# + page number" is added after the current directory path (for example, if the files currently scanned are in the abc directory of drive D, then the current directory path is d:\abc) as the source file path of the current page, and the source file path of the current page and the list of files to be transferred under the source file path of the current page are stored in the database (a current page source file path and a list of files to be transferred under the path are inserted into the database to form a record), and each record includes an id (a unique identifier of a record), a path (the path is the source file path of the current page, that is, the source path and the current page number, for example d:\abc#), a str (a JSON string corresponding to the file list in the path, and the string can also be compressed), and a distpath (backup path, specified by the user).

[0027] Step S2, obtain the list of files to be transferred scanned in the scanning phase from the database, send a comparison request to the object storage of the backup end for each source file path and its corresponding list of files to be transferred, temporarily obtain the backup end files as needed during the comparison process, add the files that have been modified on the source end and the files that are not on the backup end found in the comparison process to the list of files to be uploaded, and store them in the database according to the source end path for file transfer in the transmission phase.

[0028] Specifically, step S2 further includes: Step S200, obtaining a list of files to be transferred from a database, and assigning each source file path and its corresponding list of files to be transferred to different threads for processing; Step S201, for each source file path (path) and its corresponding list of files to be transferred, its corresponding thread sends a comparison request to the object storage of the backup end to determine whether the backup path (distpath) exists. If the backup path does not exist, all files under the current page file path (path) are added to the transfer list and saved in the database. The content of the transfer list record in the database is the same as the record saved in the list of files to be transferred in the database after the above-mentioned scan is completed, that is, it also contains id (uniquely identifies a record), path (source file path), str (json string corresponding to the file list in the path, and the string can also be compressed), and distpath (backup path).

[0029] Step S202: If the backup end path exists, loop through the list of files to be transmitted, obtain the files with the same name under the backup end path, compare the specified attributes, and add the files with different specified attributes from the files with the same name on the backup end to the list of files to be uploaded.

[0030] Specifically, when the backup path, that is, the backup directory, exists, the files in the list of files to be transferred are checked in turn, and the files with the same name under the backup path are obtained. If there is no file with the same name on the backup side, the file is added to the list of files to be uploaded. If there is a file with the same name on the backup side, the specified attributes of the two are compared, and the file with different specified attributes from the file with the same name on the backup side is added to the list of files to be uploaded. The specified attributes include file size, modification timestamp, and md5 value of the file.

[0031] In a specific embodiment of the present invention, a method for obtaining a file with the same name on a standby side is as follows: first, obtain a file with the same name from a file list on the standby side; if a file with the same name exists in the file list on the standby side, take out the corresponding file from the file list on the standby side for comparison, compare the specified attributes of the two, and add a file with different specified attributes from the file with the same name on the standby side to a list of files to be uploaded, wherein the specified attributes include file size, modification timestamp, and md5 value of the file; if the file does not exist in the file list on the standby side, if the file name is sorted after the last file in the file list on the standby side or the file list on the standby side is empty (the object storage defaults to natural sorting; if the file name is sorted after the last file in the file list on the standby side or the file list on the standby side is empty, it means that the file in the object storage on the standby side may not be recorded in the file list on the standby side), then a request is sent to the object storage on the standby side to determine whether there is a file with the same name under the path on the standby side (for The object storage provides an interface method for determining whether a file exists based on the bucket name and file name). If there is a file with the same name, a request is sent to the object storage to obtain a certain number of files after the file with the file as a mark, and the file list returned by the object storage is added to the standby file list, and the comparison is returned to continue. For example, the currently compared file is 1123.txt, and the last file in the standby file list is 1120.txt, which means that the files in the standby file list currently only reach 1120.txt, but the files after that have not been obtained. Therefore, in this case, it is necessary to take 1123.txt as a mark from the standby to add 1000 files to the standby file list for comparison (the file name in the object storage is in the form of a string and is sorted in dictionary order. The comparison is based on the Unicode value of each character in the string); after each thread has finished comparing, enter step S3.

[0032] It can be seen that the significance of the present invention is that it is not necessary to scan the entire backup directory, but only to obtain some files under the directory. The key point is to obtain a certain number of files from the backup according to a certain flag. This number can be 1000 or other values. For example: the source end has 20 files from 1 to 10 and 1002 to 1012. And the backup end has 1000000 files from 1 to 10000000. At this time, when comparing the file 1, first determine that the backup end has the file 1 (the judgment process is the process of obtaining this file) and then use the file 1 as a mark to obtain 1000 files, that is, 2 to 1001, and add these 1001 files to the backup list. When comparing the 9 files 2 to 10, you can directly take them from the backup list. When comparing 1002, it is found that there is no such file in the backup list, and 1002 is ranked after 1001, indicating that the file 1002 may exist but has not been obtained, so at this time, 1000 files are obtained with 1002 as a mark.

[0033] Step S3: Upload the files to be uploaded to the object storage according to the list of files to be uploaded.

[0034] First, obtain the transfer list of files to be uploaded in each path from the database, and assign the transfer lists of these different directories to different threads for transmission.

[0035] Preferably, if paging is performed in the scanning phase, the "#&#+page number" at the end of the source path needs to be removed in the transmission phase.

[0036] Preferably, data transfer between the scanning phase, the comparison phase, and the transmission phase can use a lightweight database, such as sqlite, mongodb.

[0037] Figure 2 This is a system architecture diagram of a file backup device based on object storage in the present invention. Figure 2 As shown, the present invention provides a file backup device based on object storage, comprising: The scanning module 20 is used to scan the source directory, scan the files in the directory to be backed up, and add the files to be backed up to the list of files to be transferred in the corresponding directory.

[0038] The scanning module 20 is specifically used for: Scan the source directory according to the list of directories to be scanned, determine whether the scanned directory needs to be backed up, and if it needs to be backed up, further determine whether it has subdirectories or files.

[0039] Specifically, a list of directories to be scanned is established. For example, the root directory is first added to the list of directories to be scanned, and the source directory is scanned according to the list of directories to be scanned. When a directory is scanned, it is determined whether the scanned directory needs to be backed up. If not, the scanning continues. If the currently scanned directory needs to be backed up, it is further determined whether there are subdirectories or files under the current directory.

[0040] If there is a file in the current directory, determine whether the file needs to be backed up. If it needs to be backed up, record the file name, file size, timestamp and other related attributes, and add the file to the list of files to be transferred in the current directory.

[0041] If there are subdirectories under the current directory, determine whether the subdirectory meets the scanning conditions. If so, add it to the list of directories to be scanned. Because customers sometimes do not want to back up all subdirectories under a directory, some subdirectories may not need to be backed up, so it is necessary to determine whether each subdirectory meets the scanning conditions (for example, users can set excluded subdirectories), that is, whether it needs to be backed up. If it needs to be backed up, add it to the list of directories to be scanned. When the current directory is scanned, the list of files to be transferred in the current directory is stored in the database, and the process returns to the above process of switching directories for scanning according to the list of directories to be scanned.

[0042] Assuming that the current directory is d:\abc and has been scanned, the list of files to be transferred in the current directory records all the files that need to be backed up in the current directory, and the current directory and its list of files to be transferred are stored in the database. The records stored in the database include id (a unique identifier for a record), path (source file path, that is, the current directory path), str (a json string corresponding to the list of files to be transferred in the path, and the string can also be compressed), and distpath (backup path, configured by the user). In a specific embodiment of the present invention, the user only needs to configure the outermost paths of the source and backup ends. After the source is scanned, the subdirectory of the source can be mapped to the backup end. For example: the source path is d:\abc, the backup end is dtotest: / bcd / , and there are subdirectories b, d, and f under drive d abc. When the list of files to be transferred corresponding to the path d:\abc\b is inserted into the database, its distpath can be automatically mapped to dtotest: / bcd / b.

[0043] Preferably, when the number of files to be backed up scanned in the current directory exceeds a preset threshold (for example, 1000), "#&# + page number" is added after the current directory path (for example, if the files currently scanned are in the abc directory of drive D, then the current directory path is d:\abc) as the source file path of the current page, and the source file path of the current page and the list of files to be transferred under the source file path of the current page are stored in the database (a current page source file path and a list of files to be transferred under the path are inserted into the database to form a record), and each record includes an id (a unique identifier of a record), a path (the path is the source file path of the current page, that is, the source path and the current page number, for example d:\abc#), a str (a JSON string corresponding to the file list in the path, and the string can also be compressed), and a distpath (backup path, specified by the user).

[0044] The comparison module 21 is used to obtain the list of files to be transferred scanned in the scanning phase from the database, send a comparison request to the object storage of the backup end for each source end file path and its corresponding list of files to be transferred, temporarily obtain the backup end files as needed during the comparison process, add the files that have been modified on the source end and the files that are not on the backup end found in the comparison process to the list of files to be uploaded, and store them in the database according to the source end path for file transfer in the transmission phase.

[0045] The comparison module 21 is specifically used for: Get the list of files to be transferred from the database, and assign each source file path and its corresponding list of files to be transferred to different threads for processing; For each source file path (path) and its corresponding list of files to be transferred, its corresponding thread sends a comparison request to the standby object storage to determine whether the standby path (distpath) exists. If the standby path does not exist, all files under the current page file path (path) are added to the transfer list and saved in the database. The content of the transfer list record in the database is the same as the record saved in the list of files to be transferred in the database after the above scan, that is, it also contains id (uniquely identifies a record), path (source file path), str (json string corresponding to the file list in the path, and the string can also be compressed), and distpath (standby path).

[0046] If the backup end path exists, the list of files to be transferred is traversed in a loop, and files with the same name under the backup end path are obtained for comparison of specified attributes, and files with different specified attributes from the backup end files with the same name are added to the list of files to be uploaded.

[0047] Specifically, when the backup path, that is, the backup directory, exists, the files in the list of files to be transferred are checked in turn, and the files with the same name under the backup path are obtained. If there is no file with the same name on the backup side, the file is added to the list of files to be uploaded. If there is a file with the same name on the backup side, the specified attributes of the two are compared, and the file with different specified attributes from the file with the same name on the backup side is added to the list of files to be uploaded. The specified attributes include file size, modification timestamp, and md5 value of the file.

[0048] In a specific embodiment of the present invention, a method for obtaining a file with the same name on a standby side is as follows: first, obtain a file with the same name from a file list on the standby side; if a file with the same name exists in the file list on the standby side, take out the corresponding file from the file list on the standby side for comparison, compare the specified attributes of the two, and add a file with different specified attributes from the file with the same name on the standby side to a list of files to be uploaded, wherein the specified attributes include file size, modification timestamp, and md5 value of the file; if the file does not exist in the file list on the standby side, if the file name is sorted after the last file in the file list on the standby side or the file list on the standby side is empty (the object storage defaults to natural sorting; if the file name is sorted after the last file in the file list on the standby side or the file list on the standby side is empty, it means that the file in the object storage on the standby side may not be recorded in the file list on the standby side), then a request is sent to the object storage on the standby side to determine whether there is a file with the same name under the path on the standby side (object storage). The storage provides an interface method for determining whether a file exists based on the bucket name and file name). If there is a file with the same name, a request is sent to the object storage to obtain a certain number of files after the file using the file as a marker, and the file list returned by the object storage is added to the backup file list, and the list is returned for further comparison. For example, the currently compared file is 1123.txt, and the last file in the backup file list is 1120.txt, which means that the files in the backup file list currently only reach 1120.txt, but the files after that have not been obtained. Therefore, in this case, it is necessary to use 1123.txt as a marker to obtain 1000 files from the backup and add them to the backup file list for comparison (the file name in the object storage is in the form of a string and is sorted in dictionary order. The comparison is based on the Unicode value of each character in the string); after each thread has finished comparing, enter the upload module 22.

[0049] The upload module 22 is used to upload the files to be uploaded to the standby object storage according to the list of files to be uploaded.

[0050] First, obtain the list of files to be uploaded for each path (directory) from the database, and assign the list of files to be uploaded in these different directories to different threads for transmission.

[0051] Preferably, if paging is performed in the scanning phase, the "#&#+page number" at the end of the source path needs to be removed in the transmission phase.

[0052] Preferably, data transfer between the scanning phase, the comparison phase, and the transmission phase can use a lightweight database, such as sqlite, mongodb. Example

[0053] In this embodiment, the process of a file backup method based on object storage is as follows: The first step is to scan the source directory.

[0054] The main purpose of scanning the source directory is to obtain a list of files that need to be compared. When a directory is scanned, determine whether the directory needs to be backed up. If it needs to be backed up, scan the files in it. Every time a file record is scanned, determine whether the file is a file that needs to be transferred. If the file needs to be transferred, record the file name, file size, timestamp and other related attributes, and add the file to the list of files to be transferred in the directory. When the number of files scanned in the directory exceeds 1000, add "#&# + page number" after the file path to store the file path and the file in the database. Each record includes id (unique mark of a record), path (path), str (json string corresponding to the file list in the path, the string can also be compressed), distpath (backup path). Here, the user only needs to configure the outermost path of the source and backup ends. After the source is scanned, the subdirectory of the source end can be mapped to the backup end, such as Figure 3 shown.

[0055] The second step is to compare the source and the backup end. The list of files to be transferred scanned in the scanning phase is retrieved from the database. Each path and its corresponding list of files to be transferred are assigned to different threads for comparison. When comparing, a request is first made to the object storage to determine whether the backup path exists. If the backup path does not exist, all files under the path are added to the list of files to be uploaded and saved in the database. The database record content contains id, path, str, and distpath, just like the records saved after the scan. If the backup directory exists, the source file list is looped through and the files with the same name on the backup end are taken for comparison of the specified attributes. If there is no file with the same name on the backup end in the case of the backup directory, the file is added to the list of files to be uploaded. If there is a file with the same name on the backup end and the file has different specified attributes, it is also added to the list of files to be uploaded. The method for obtaining files with the same name on the backup end is as follows: first, obtain files with the same name from the backup file list. If the file exists in the backup file list, the file is taken out for comparison. If the file does not exist in the backup file list, if the file name is sorted after the last file in the backup file list or the backup file list is empty, a request is sent to the object storage to determine whether there is a file with the same name in the corresponding path on the backup side. If there is a file with the same name, a request is sent to the object storage to obtain the 1000 files after the file with the file as the mark (the object storage is naturally sorted by default, and the file list returned by the object storage is added to the backup file list, and the file is returned, such as Figure 4 shown.

[0056] The third step is file upload. This stage mainly uploads the files to be backed up to the object storage. First, query the list of files to be uploaded from the database, and assign the file lists in different directories to different threads for transmission. Since paging is performed during the scanning stage, the "#&#+page number" at the end of the source path needs to be removed during the transmission stage.

[0057] The above embodiments are only illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Any person skilled in the art may modify and change the above embodiments without violating the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be as set forth in the claims.

Claims

1. A file backup method based on object storage, comprising the following steps: Step S1, scanning the source directory, scanning the files in the directory to be backed up, adding the files to be backed up to the list of files to be transferred in the corresponding directory and storing them in the database; Step S2, obtaining the list of files to be transmitted from the database, sending a comparison request to the standby object storage for each path and its corresponding list of files to be transmitted, obtaining some files on the standby side for comparison according to the file names on the source side, adding the files that have been modified on the source side and the files that are not on the standby side found in the comparison process to the list of files to be uploaded, and storing them in the database according to the source path for file transmission in the transmission stage; Step S3: uploading the files to be backed up to the backup object storage according to the list of files to be uploaded.

2. The file backup method based on object storage according to claim 1, characterized in that: Step S1 further comprises: Step S100, scanning the source directory according to the list of directories to be scanned, determining whether the scanned directory needs to be backed up, and if it needs to be backed up, further determining whether it has subdirectories or files; Step S101, if there is a file in the current directory, determine whether the file needs to be backed up. If it needs to be backed up, record the relevant attributes of the file and add it to the list of files to be transferred corresponding to the current directory; Step S102, if there is a subdirectory under the current directory, determine whether the subdirectory meets the scanning condition, and if so, add it to the list of directories to be scanned; Step S103, when the current directory is scanned, the list of files to be transferred in the current directory is stored in the database, and the process returns to step S101 to switch directories for scanning according to the list of directories to be scanned.

3. The file backup method based on object storage according to claim 2, characterized in that: Each record stored in the database includes a unique identifier (ID), a source path (path), a JSON string (str) corresponding to the list of files to be transferred of the source path, and a backup path (distpath).

4. The file backup method based on object storage according to claim 3, characterized in that: When the number of files scanned in the current directory exceeds a preset threshold, after adding paging information after the source end path, the source end path with the paging information added and the corresponding list of files to be transferred are stored in the database.

5. The file backup method based on object storage according to claim 3, characterized in that: Step S2 further comprises: Step S200, obtaining a list of files to be transferred from a database, and assigning each source end path and its corresponding list of files to be transferred to different threads for processing; Step S201: for each source end path and its corresponding list of files to be transferred, its corresponding thread sends a comparison request to the backup end object storage to determine whether the backup end path exists. If the backup end path does not exist, all files under the source end path are added to the list of files to be uploaded and saved in the database. Step S202: if the backup end path exists, loop through the list of files to be transmitted of the source end path, obtain files with the same name on the backup end for comparison of specified attributes, and add files with different specified attributes from the files with the same name on the backup end to the list of files to be uploaded.

6. The file backup method based on object storage according to claim 5, characterized in that: In step S202, when there is a backup path, the files in the list of files to be transferred are checked in turn to obtain the files with the same name on the backup side. If there is no file with the same name on the backup side, the file is added to the list of files to be uploaded. If there is a file with the same name on the backup side, the specified attributes of the two are compared, and the file with different specified attributes from the file with the same name on the backup side is added to the list of files to be uploaded.

7. The file backup method based on object storage according to claim 6, characterized in that: The specified attributes include file size, modification timestamp, and MD5 value of the file.

8. The file backup method based on object storage according to claim 6, characterized in that: In step S202, the method for obtaining the same-name file on the backup end is as follows: Obtain the file with the same name from the backup file list. If there is a file with the same name in the backup file list, take out the corresponding file for comparison, compare the specified attributes of the two, and add the file with different specified attributes from the backup file with the same name to the transfer list; If there is no file with the same name in the backup file list, if the file name is sorted after the last file in the backup file list or the backup file list is empty, a request is sent to the object storage to determine whether there is a file with the same name in the corresponding path on the backup side. If there is a file with the same name, a request is sent to the object storage to obtain a preset threshold number of files after the file using the file as a marker, and the file list returned by the object storage is added to the backup file list, and the file is returned.

9. A file backup method based on object storage as claimed in claim 8, characterized in that : In step S3, a list of files to be uploaded of each source path is obtained from the database, and the lists of files to be uploaded of different directories are allocated to different threads for transmission.

10. A file backup device based on object storage, comprising: The scanning module is used to scan the source directory, scan the files in the directory to be backed up, add the files to be backed up to the list of files to be transferred in the corresponding directory and store them in the database; The comparison module obtains the list of files to be transferred from the database, sends a comparison request to the standby object storage for each path and its corresponding list of files to be transferred, obtains some files on the standby side according to the file names on the source side for comparison, adds the files that have been modified on the source side and the files that are not on the standby side found in the comparison process to the list of files to be uploaded, and stores them in the database according to the source path for file transfer in the transmission stage; The upload module uploads the files to be backed up to the backup object storage according to the list of files to be uploaded.

Images