An index structure, a file backup method, a file recovery method, and a system

Abstract

This invention discloses an index structure, a file backup method, a file recovery method, and a system, belonging to the field of file backup and recovery technology. It includes: an index file for recording the location of files in the index and information about the backed-up files; a directory entry file for recording the names of the backed-up directories and files, and whose absolute paths and all subdirectories and files under this directory can be obtained based on their positions in the directory entry file; and a data file for storing the data of the files in the backup directory. This index structure, file backup method, file recovery method, and system consume few system resources, have a fast response time, and require only a few I / O operations when querying and recovering directories or files. The number of I / O operations does not increase linearly with the number of directories or files. It also has strong scalability, supporting network storage, compression, deduplication, encryption, and other extensions. It has no requirements for the operating environment, is easy to use, and reduces development and maintenance difficulty.

Description

Technical Field

[0001] This invention belongs to the field of file backup and recovery technology, specifically relating to an index structure, a file backup method, a file recovery method, and a system. Background Technology

[0002] Existing mainstream massive file backup systems store file information in a third-party database. During backup, file information is inserted into the database, and records are searched from the database when searching for and restoring files. When backing up files, if a large number of small files are encountered, they are often merged into a large file to generate a backup set, thereby saving IO overhead when reading and writing data.

[0003] Mainstream massive file backup systems typically use third-party databases, such as MongoDB, for index data management. In this approach, the third-party database requires certain resources, including memory, disk space, and CPU resources. Using a third-party database requires learning, which increases the difficulty of development and subsequent maintenance. When problems occur, if the user is not proficient in the database, technical support from the original manufacturer may be needed. Mainstream databases are not optimized for scenarios involving storing file information, resulting in low management efficiency. Therefore, it is necessary to develop a new index structure, file backup method, file recovery method, and system to solve the existing problems. Summary of the Invention

[0004] The purpose of this invention is to provide an index structure, a file backup method, a file recovery method, and a system to solve the problem that massive file backups require reliance on third-party databases for index data management.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an index structure, comprising: an index file, used to record the location of files in the index and backup file information;

[0006] A directory entry file records the names of backed-up directories and files, and allows you to retrieve their absolute paths and all subdirectories and files within that directory based on their position in the directory entry file; and

[0007] Data files are used to store data from files in the backup directory;

[0008] The index file includes:

[0009] File nodes are used to perform hash calculations on the absolute paths of directories or files to obtain the node number inode, which is then used as a global file identifier.

[0010] The hash array and hash linked list are used to obtain the hash array and hash linked list by performing a modulo operation based on the node number inode;

[0011] The header file is used to record information about the directory entry files and data files associated with the index file, the number of file nodes, and file groupings.

[0012] Preferably, when recording the directory entry file, the directory level of the backup object in the absolute path is set as the depth, and the depth is sorted according to the following strategy:

[0013] a. Files and directories in a directory are listed after the parent directory;

[0014] b. Object directories under the same path are arranged before files, and inode values ​​are arranged in ascending order.

[0015] This invention also provides a file backup method based on an index structure, characterized in that it includes:

[0016] Scan the target directory;

[0017] Generate file nodes and directory nodes;

[0018] Take the modulo of the hash and generate a hash linked list, and generate an index file and a directory entry file;

[0019] File grouping;

[0020] Back up the data files and record the data information in the corresponding file nodes of the index file;

[0021] Backup the index files and directory files.

[0022] Preferably, the generated file nodes and directory nodes include:

[0023] After scanning all subdirectories and subfiles in the target path, the absolute path of the scanned directory or file is hashed to obtain the globally unique identifier inode. The corresponding file information is recorded in the index node. The file node part is generated according to the scanning order, and the inode and file or directory name are recorded. The directory node is generated in the same order as the file node, so that the positions of the index node and the directory node correspond one-to-one when the generated records are read sequentially.

[0024] Preferably, the step of taking the modulo of the hash and generating a hash linked list includes:

[0025] The hash value of each index node (inode) is moduloed to obtain the modulo value. All moduli form a hash array. Inodes with the same modulo form a hash linked list under this modulo. The linked list records the inode value and the offset of the corresponding file node in the entire index file.

[0026] The generation of the index file and directory entry file includes:

[0027] The hash array, hash list, and file nodes are merged in sequence, with space reserved at the beginning to record the positions of directory files and data files, to generate an index file. Directory entry nodes are then merged to generate a directory entry file.

[0028] Preferably, the backup data file includes:

[0029] After obtaining the inode information and backup target path of the file, the scanned data file is backed up, the file is uploaded to the backup medium, the uploaded file is merged into one or more data files in the backup medium, and the starting offset of the data file is returned.

[0030] The data information recorded in the index node includes:

[0031] Once the data file backup is successful, the offset bit in the backup file is written back into the corresponding inode. The information of the backup media and backup file is recorded in the reserved space in the header. The corresponding record is marked in the inode. The record is read by reading the mark in the inode, and the data within the specified range in the backup file is read to restore the original data file.

[0032] This invention also provides a file recovery method based on an index structure, comprising:

[0033] Read the index file;

[0034] Calculate the target inode to be recovered;

[0035] Traverse the hash linked list to locate the index node, traverse the index node and its corresponding directory node, and traverse all subfiles or subdirectories;

[0036] Download and restore the data file.

[0037] Preferably, the calculation to recover the target inode includes:

[0038] The absolute address of the input recovery target is hashed to obtain the inode value. The modulo of the inode value is then taken to find the corresponding hash array.

[0039] Preferably, traversing the hash linked list to locate the index node includes:

[0040] Traverse the hash linked list under the hash array, find the corresponding inode record, and jump to the position of the corresponding index node in the index file according to the offset of the record;

[0041] The traversal of index nodes and corresponding directory nodes includes:

[0042] Starting from the index node position of the file determined during the traversal scan, the index node records are traversed sequentially. At the same time, based on the position of the index node and the directory file information of the record, the corresponding directory node record is found and matched with the index node, and the sequential traversal is performed.

[0043] The traversal of all subfiles or subdirectories includes:

[0044] During the traversal, the directory depth information recorded in the directory nodes is read, and all subdirectories and subfiles in the recovery path are scanned. When the depth is lower than the depth of the directory at the traversal starting point, it indicates that the traversal is complete, the traversal is exited, and all the traversed information is recorded for the recovery of data files.

[0045] The present invention also provides a file backup and recovery system based on an index structure, comprising:

[0046] The index data management module is used for writing, searching, and reading index files;

[0047] The data backup module is used to scan files, generate indexes, and call the interface of the source data management module to add indexes.

[0048] The data recovery module is used to call the interface of the source data management module to find the index and restore it to the target path.

[0049] The technical effects and advantages of this invention are as follows: The index structure, file backup method, file recovery method, and system consume minimal system resources. Through the design of the index and directory storage, the source code size is only about 100KB, and the index file generated by backing up millions of small files does not exceed 100MB. During reading, it can be easily loaded into the memory of mainstream devices. These resources are only consumed when calling relevant system functions, while third-party databases usually cannot be closed. Furthermore, this solution has a fast response time, requiring only a few I / O operations when querying and recovering directories or files. The number of I / O operations does not increase linearly with the number of directories or files. It also has strong scalability, supporting network storage, compression, deduplication, encryption, and other extensions. It has no requirements for the operating environment, is easy to use, and reduces the difficulty of development and maintenance. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the composition of the index structure of the present invention;

[0051] Figure 2 This is a flowchart of the file backup method of the present invention;

[0052] Figure 3 This is a flowchart of the file recovery method of the present invention;

[0053] Figure 4 This is a schematic diagram of the system framework of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] This invention provides, for example Figure 1 The index structure shown includes an index file, a directory entry file, and a data file;

[0056] The index file is used to record the location of various files in the index and the backup file information. Structurally, it consists of four parts: file nodes, hash array, hash linked list and header file, forming an indirect index.

[0057] File node: The absolute path of a directory or file is hashed to obtain the node number inode, which is used as the global file identifier. The inode and file information together constitute the file node. This record is the position of the corresponding record in the directory entry file and the offset and file size in the data file.

[0058] Hash array and hash list: The hash array and hash list are obtained by performing a modulo operation on the node number inode. The hash value of each index node inode is moduloed to obtain the modulo value. Inodes with the same modulo form a hash list. The inode and the offset of the corresponding index node record in the entire index file are recorded in the list.

[0059] Header file: The header file records information about the directory entry files and data files associated with the records in this index file, the number of file nodes, and file grouping. During I / O addressing, the number of read / write heads on a hard drive is limited. Continuously increasing I / O concurrency will not improve efficiency; instead, it will cause I / O contention and reduce efficiency. Therefore, in this solution, file data in the same area on the hard drive will be grouped into one task as much as possible, and the task will be assigned to different threads to improve file read / write efficiency.

[0060] The directory entry file records the names of the backed-up directories and files, and their absolute paths and all subdirectories and files within that directory can be retrieved based on their position in the file. The directory entry record sets the directory level of the backup object within the absolute path as its depth, and is sorted according to the following strategy:

[0061] Files and directories in a directory are listed after their parent directories.

[0062] Objects under the same path are listed in order of directory first, file second, and inode value smaller first and larger last. Based on this rule, the tree structure of the file directory is converted into a singly linked list. The addressing characteristics of commonly used storage media are that random reads will cause additional IO overhead, while sequential reads only require a few IO addressings. When searching, the sorted linked list is used to always perform sequential reads, which significantly improves the speed. The size of the directory entry node can be changed to be compatible with the maximum upper limit of file name length in different systems.

[0063] The data file stores the data of the files in the backup directory. This file is merged into one or more large files and records its corresponding offset. These large files are called data files. When searching for the target file from a large number of small files, a lot of I / O operations are required to find the file and read the data. However, when they are merged into a large file, the file read and write efficiency can be greatly improved.

[0064] The present invention further provides, for example, Figure 2 The file backup method based on an index structure shown includes the following steps:

[0065] Scan the target directory.

[0066] The backup directory is scanned according to sorting rules to read various information about directories, files, or symbolic links;

[0067] Generate file nodes and directory nodes.

[0068] After scanning all subdirectories and subfiles in the target path, the absolute path of the scanned directory or file is hashed to obtain the globally unique identifier inode of the record. The corresponding file information is recorded in the index node. The file node part is generated according to the scanning order, and the inode and file or directory name are recorded. The directory node is generated in the same order as the file node. When the generated records are read sequentially, the positions of the index node and the directory node will correspond one by one, without the need for additional fields to record the correspondence.

[0069] Take the modulo of the hash and generate a hash linked list.

[0070] The hash value (inode) of each index node is moduloed to obtain the modulo value. All modulo values ​​form a hash array. Inodes with the same modulo form a hash linked list for that modulo. The linked list records the inode value and the offset of the corresponding file node in the entire index file. The target file node can be quickly located and the data in the file node can be read through the hash linked list.

[0071] Generate index files and directory files.

[0072] The hash array, hash linked list, and file nodes are merged in order, and a certain amount of space is reserved at the beginning to record the positions of the directory file and data file. Then, an index file is generated, and the directory entry nodes are merged to generate a directory entry file. Each file node in the index file corresponds to a directory entry in the directory entry file.

[0073] File grouping;

[0074] Based on the location of the target files on the hard drive, group them and back up the data files in groups.

[0075] Backup data files,

[0076] After obtaining the inode information and target path of the file, the scanned data file is backed up; the file is uploaded to the backup medium, and these files are merged into one or more data files in the backup medium, and the starting offset in the data file is returned;

[0077] Data information is recorded in the index node.

[0078] When a data file is successfully backed up, the offset in the backup file is written back in the corresponding inode. The information of the backup media and backup file is recorded in the reserved space in the header. The corresponding record is marked in the inode. The record can be read through the mark in the inode, and then the data of the specified range in the backup file can be read to restore the original data file.

[0079] Backup the index files and directory files.

[0080] The present invention further provides, for example, Figure 3 The following is a file recovery method based on an index structure, comprising the following steps:

[0081] Read the index file.

[0082] The corresponding index is located based on the recorded index file information and backup media information, and the index file is read. Depending on the actual size of the index file, it is possible to choose whether to load the index file into memory at once.

[0083] Calculate the target inode to be recovered.

[0084] The absolute address of the input recovery target is hashed to obtain the inode value. The modulo of the inode value is then taken to find the corresponding hash array.

[0085] Traverse the hash linked list to locate the index node.

[0086] Traverse the hash linked list under the hash array to find the corresponding inode record. Jump to the corresponding inode location in the index file based on the offset of the record;

[0087] Simultaneously traverse the index nodes and their corresponding directory nodes.

[0088] Starting from the inode position of the file determined during the traversal scan, the inode records are traversed sequentially. Simultaneously, based on the inode position and the directory file information of the record, the corresponding directory node record is found and matched with the inode, and then traversed sequentially.

[0089] Traverse all subfiles or subdirectories.

[0090] During the traversal, the directory depth information recorded in the directory nodes is read. The goal of the traversal is to scan all subdirectories and subfiles in the recovery path. When the depth is lower than the directory depth at the traversal starting point, it indicates that the traversal is complete, and the traversal is exited. All the traversed information is recorded for use in recovering the data files.

[0091] Download and restore the data files.

[0092] Based on the data file information recorded in the index file and the offsets recorded in the inodes, download the data of the specified segment from the backup file. Restore the file according to the location and name recorded in the directory file.

[0093] The present invention further provides, for example, Figure 4 The file backup and recovery system based on an index structure shown includes: a lightweight index structure and a backup and recovery system for massive small files implemented based on this index; including an index data management module, a data backup module, and a data recovery module;

[0094] The index data management module is used to write, search, and read index files;

[0095] The data backup module is used to scan files, generate indexes, and call the interface of the source data management module to add indexes;

[0096] The data recovery module calls the interface of the source data management module to find the index and restore it to the target path;

[0097] This index structure, file backup method, file recovery method, and system consume minimal system resources. Through the design of the index and directory storage, the source code is only about 100KB in size, and the index file generated by backing up millions of small files does not exceed 100MB. During reading, it can be easily loaded into the memory of mainstream devices. These resources are only consumed when calling relevant system functions, whereas third-party databases usually cannot be shut down. Furthermore, this solution has a fast response time, requiring only a few I / O operations when querying and recovering directories or files. The number of I / O operations does not increase linearly with the number of directories or files. It is also highly scalable, supporting network storage, compression, deduplication, encryption, and other extensions. It has no requirements for the operating environment, is easy to use, and reduces the difficulty of development and maintenance.

[0098] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A file backup method based on an index structure, characterized in that: The index structure includes: Index files are used to record the location of files in the index and information about backup files; A directory entry file records the names of backed-up directories and files, and allows you to retrieve their absolute paths and all subdirectories and files within that directory based on their position in the directory entry file; and Data files are used to store data from files in the backup directory; The index file includes: File nodes are used to perform hash calculations on the absolute paths of directories or files to obtain the node number inode, which is then used as a global file identifier. The hash array and hash linked list are used to obtain the hash array and hash linked list by performing a modulo operation based on the node number inode; The header file is used to record information about the directory entry files and data files associated with the index file, the number of file nodes, and file groupings; When recording directory entry files, the directory level of the backup object in the absolute path is set as the depth, and the depth is sorted according to the following strategy: Files and directories in a directory are listed after their parent directories; directories containing objects on the same path are listed before their files, and inode values ​​are sorted in ascending order. The method includes: Scan the target directory; Generate file nodes and directory nodes; Take the modulo of the hash and generate a hash linked list, and generate an index file and a directory entry file; File grouping; Back up the data files and record the data information in the corresponding file nodes of the index file; Backup index files and directory files; The step of taking the modulo of the hash and generating a hash linked list includes: The hash value of each index node (inode) is moduloed to obtain the modulo value. All moduli form a hash array, and inodes with the same modulo form a hash linked list under this modulo. The linked list records the inode value and the offset of the file node in the entire index file. The generation of the index file and directory entry file includes: The hash array, hash list, and file nodes are merged in sequence, with space reserved at the beginning to record the positions of directory files and data files, to generate an index file. Directory entry nodes are then merged to generate a directory entry file.

2. The file backup method based on an index structure according to claim 1, characterized in that: The generated file nodes and directory nodes include: After scanning all subdirectories and subfiles in the target path, the absolute path of the scanned directory or file is hashed to obtain the globally unique identifier inode. The corresponding file information is recorded in the index node. The file node part is generated according to the scanning order, and the inode and file or directory name are recorded. The directory node is generated in the same order as the file node, so that the generated records correspond to the positions of the index node and the directory node when reading sequentially.

3. The file backup method based on an index structure according to claim 1, characterized in that: The backup data files include: After obtaining the inode information and backup target path of the file, the scanned data file is backed up, the file is uploaded to the backup medium, the uploaded file is merged into one or more data files in the backup medium, and the starting offset of the data file is returned. The file node record data information in the index file includes: Once the data file backup is successful, the offset bit in the backup file is written back into the corresponding inode. The information of the backup media and backup file is recorded in the reserved space in the header. The corresponding record is marked in the inode. The record is read by reading the mark in the inode, and the data within the specified range in the backup file is read to restore the original data file.

4. A file recovery method based on an index structure, applied to the file backup method based on an index structure as described in claim 1, characterized in that: include: Read the index file; Calculate the target inode to be recovered; Traverse the hash linked list to locate the index node, traverse the index node and its corresponding directory node, and traverse all subfiles or subdirectories; Download and restore the data file.

5. The file recovery method based on an index structure according to claim 4, characterized in that: The target inode for computational recovery includes: The absolute address of the input recovery target is hashed to obtain the inode value. The modulo of the inode value is then taken to find the corresponding hash array.

6. The file recovery method based on an index structure according to claim 5, characterized in that: The process of traversing the hash linked list to locate the index node includes: Traverse the hash linked list under the hash array, find the corresponding inode record, and jump to the position of the corresponding index node in the index file according to the offset of the record; The traversal of index nodes and corresponding directory nodes includes: Starting from the file node position determined during the traversal scan, the index node records are traversed sequentially. At the same time, based on the position of the index node and the directory file information of the record, the corresponding directory node record is found and matched with the index node, and then the sequential traversal is performed. The traversal of all subfiles or subdirectories includes: During the traversal, the directory depth information recorded in the directory nodes is read, and all subdirectories and subfiles in the recovery path are scanned. When the depth is lower than the depth of the directory at the traversal starting point, it indicates that the traversal is complete, the traversal is exited, and all the traversed information is recorded for the recovery of data files.

7. A system for implementing the file backup method based on an index structure as described in claim 1, characterized in that: The system includes: The index data management module is used for writing, searching, and reading index files; The data backup module is used to scan files, generate indexes, and call the interface of the source data management module to add indexes. The data recovery module is used to call the interface of the source data management module to find the index and restore it to the target path.

Images