File backup method and device, equipment and storage medium

By employing parallel processing technology and a multi-threaded architecture, the inefficiency of traditional tar tools in backing up large datasets has been resolved, resulting in a highly efficient file backup method that improves backup speed and system scalability.

CN119025341APending Publication Date: 2024-11-26SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411275427.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Traditional tar file archiving tools are inefficient when dealing with large datasets, lack support for multi-threaded processing, and cannot effectively speed up the backup process.

Method used

Parallel processing technology is employed to determine file types by scanning the file system, generate target operation handles, and add files to the corresponding queues. A multi-threaded architecture is used to package files in parallel, and a diverse data encapsulation structure is designed to fully utilize the capabilities of modern multi-core processors.

Benefits of technology

It significantly reduces the time required for backup, enhances the scalability and flexibility of the system, and meets the needs of large-scale data backup.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025341A_ABST
    Figure CN119025341A_ABST
Patent Text Reader

Abstract

The invention discloses a file backup method and device, equipment and a storage medium, and relates to the technical field of information, and the method comprises the following steps: scanning a plurality of files to be backed up in a file system; determining a file type corresponding to the current file to be backed up in the plurality of files to be backed up; generating a target operation handle corresponding to the file type, and adding the current file to be backed up to a target operation handle queue corresponding to the target operation handle based on the target operation handle; taking the next file to be backed up as the current file to be backed up, and skipping to the step of traversing the plurality of files to be backed up to determine the file type corresponding to the current file to be backed up in the plurality of files to be backed up until the plurality of files to be backed up are all added to the corresponding target operation handle queue; and according to the determined file backup type, packaging the to-be-backed-up files in the plurality of target operation handle queues in parallel, and backing up the plurality of obtained packaged data packets. Therefore, the time required for backup can be greatly shortened by utilizing a parallel processing technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a file backup method, device, equipment and storage medium. Background Art

[0002] In the field of modern data management and storage backup, the traditional tar (tape archive, a compression and packaging tool on Unix and Unix-like systems) file archiving tool is well-known for its long history and wide application. It can effectively integrate a large number of files into a single archive file for easy storage and maintenance. However, tar was originally designed for the tape storage era. Faced with the rapid expansion of contemporary data scale, especially in the processing of massive small files, its performance bottleneck has gradually emerged. The working mode of tar requires that all files to be backed up be traversed in advance before packaging can begin, which means that the time consumption of the entire backup process directly adds up to the time required for file traversal and actual packaging, resulting in inefficiency when processing large data sets. In addition, the lack of support for multi-threaded processing limits its performance potential in a multi-core processor environment and cannot effectively accelerate the backup process.

[0003] It can be seen that how to implement a more efficient backup solution to cope with the growing demand for data backup and recovery is an issue that needs to be addressed urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a file backup method, device, equipment and storage medium, which can use parallel processing technology to significantly shorten the time required for backup, while enhancing the scalability and flexibility of the system to meet the needs of large-scale data backup. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a file backup method, which is applied to a file system, comprising:

[0006] Scanning a file system to determine a number of files to be backed up in the file system;

[0007] Traversing the plurality of files to be backed up to determine a file type corresponding to a current file to be backed up among the plurality of files to be backed up;

[0008] Generate a target operation handle corresponding to the file type, and add the current file to be backed up to a target operation handle queue corresponding to the target operation handle based on the target operation handle;

[0009] The next file to be backed up is used as the current file to be backed up, and the process jumps to the step of traversing the plurality of files to be backed up to determine the file type corresponding to the current file to be backed up among the plurality of files to be backed up, until the plurality of files to be backed up are all added to the corresponding target operation handle queue;

[0010] The files to be backed up in a plurality of target operation handle queues are packaged in parallel according to the determined file backup type, and the obtained plurality of packaged data packets are backed up.

[0011] Optionally, traversing the plurality of files to be backed up to determine a file type corresponding to a current file to be backed up among the plurality of files to be backed up includes:

[0012] Traversing the plurality of files to be backed up, and determining the attribute value of the current file to be backed up among the plurality of files to be backed up by a first preset method;

[0013] If the attribute value is greater than a preset attribute value threshold, determining that the file type of the current file to be backed up is a hard link file;

[0014] If the attribute value is not greater than the preset attribute value threshold, detecting whether the file type of the current file to be backed up is a soft link file by a second preset method, and obtaining a first return result corresponding to the second preset method;

[0015] If the first returned result indicates that the current file to be backed up is not a soft link file, detecting whether the file type of the current file to be backed up is a directory file by a third preset method, and obtaining a second returned result corresponding to the third preset method;

[0016] If the second returned result indicates that the current file to be backed up is not a directory file, then the file type of the current file to be backed up is determined to be a regular file.

[0017] Optionally, the generating a target operation handle corresponding to the file type, and adding the current file to be backed up to a target operation handle queue corresponding to the target operation handle based on the target operation handle, includes:

[0018] Generate a target operation handle corresponding to the file type, and establish a corresponding relationship between the current file to be backed up and the target operation handle, so as to add the current file to be backed up to a target operation handle queue corresponding to the target operation handle through the target operation handle;

[0019] If the file to be backed up has been added to the target operation handle queue, destroying the target operation handle;

[0020] Among them, the target operation handle includes a hard link handle, a soft link handle, a directory handle and a file handle, and the target operation handle queue includes a hard link queue corresponding to the hard link handle, a soft link queue corresponding to the soft link handle, a directory queue corresponding to the directory handle and a file queue corresponding to the file handle.

[0021] Optionally, the step of packaging a plurality of files to be backed up in the target operation handle queue in parallel according to the determined file backup type, and backing up the obtained plurality of packaged data packets includes:

[0022] The file backup type is determined according to the received backup operation instruction, and the to-be-backed files in the target operation handle queues are packaged in parallel according to the file backup type through several preset packaging components corresponding to the file type, and the obtained packaged data packets are backed up.

[0023] Optionally, the step of packaging a plurality of files to be backed up in a plurality of target operation handle queues in parallel according to the file backup type by using a plurality of preset packaging components corresponding to the file type, and backing up the obtained packaged data packets includes:

[0024] If the file backup type is a full backup, the first backup start time is recorded, and the to-be-backed-up files in the directory queue are packaged by a preset directory packaging component based on a first preset data encapsulation structure to obtain a packaged directory data packet;

[0025] and, packaging the to-be-backed-up files in the file queue based on a second preset data encapsulation structure by a preset file packaging component to obtain a packaged file data packet;

[0026] and, packaging the to-be-backed-up files in the soft link queue based on a third preset data encapsulation structure by a preset soft link packaging component to obtain a packaged soft link data packet;

[0027] and, packaging the to-be-backed-up files in the hard link queue based on the third preset data encapsulation structure by a preset hard link packaging component to obtain a packaged hard link data packet.

[0028] Optionally, the step of packaging a plurality of files to be backed up in the target operation handle queue in parallel according to the determined file backup type, and backing up the obtained plurality of packaged data packets includes:

[0029] If the file backup type is incremental backup, then record the second backup start time, and determine the first backup start time and modification time corresponding to the plurality of files to be backed up;

[0030] If the plurality of files to be backed up do not have modification time, or the modification time is not greater than the first backup start time, then the incremental backup operation is stopped;

[0031] If the modification time is greater than the first backup start time, incremental backup is performed on the plurality of files to be backed up, and the second backup start time is used as the first backup start time for next determination.

[0032] Optionally, after the step of packaging the to-be-backed files in the plurality of target operation handle queues in parallel according to the determined file backup type and backing up the plurality of packaged data packets obtained, the step further includes:

[0033] If a data recovery instruction is received, the plurality of packaged data packets are read and parsed, so as to perform a data recovery operation through the plurality of to-be-recovered files corresponding to the plurality of packaged data packets.

[0034] In a second aspect, the present application discloses a file backup device, which is applied to a file system, comprising:

[0035] A file scanning module, used for scanning the file system to determine a number of files to be backed up in the file system;

[0036] A file type determination module, used for traversing the plurality of files to be backed up to determine a file type corresponding to a current file to be backed up among the plurality of files to be backed up;

[0037] A file adding module, used for generating a target operation handle corresponding to the file type, so as to add the current file to be backed up to a target operation handle queue corresponding to the target operation handle based on the target operation handle;

[0038] a step jump module, used for taking the next file to be backed up as the current file to be backed up, and jumping to the step of traversing the plurality of files to be backed up to determine the file type corresponding to the current file to be backed up among the plurality of files to be backed up, until the plurality of files to be backed up are all added to the corresponding target operation handle queue;

[0039] The data backup module is used to package several files to be backed up in the target operation handle queue in parallel according to the determined file backup type, and to back up the obtained several packaged data packets.

[0040] In a third aspect, the present application discloses an electronic device, comprising:

[0041] Memory, used to store computer programs;

[0042] The processor is used to execute the computer program to implement the file backup method as described above.

[0043] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned file backup method when executed by a processor.

[0044] In the present application, it is first necessary to scan the file system to determine a number of files to be backed up in the file system; traverse the number of files to be backed up to determine the file type corresponding to the current file to be backed up among the number of files to be backed up; generate a target operation handle corresponding to the file type, and add the current file to be backed up to the target operation handle queue corresponding to the target operation handle based on the target operation handle; take the next file to be backed up as the current file to be backed up, and jump to the step of traversing the number of files to be backed up to determine the file type corresponding to the current file to be backed up among the number of files to be backed up, until the number of files to be backed up are all added to the corresponding target operation handle queue; package the files to be backed up in the target operation handle queue in parallel according to the determined file backup type, and back up the obtained number of packaged data packets. It can be seen that, through the method of the present application, several files to be backed up in the file system can be scanned and the file types corresponding to the several files to be backed up can be determined, and then a target operation handle corresponding to the file type can be generated, and the files to be backed up can be added to the target operation handle queue corresponding to the target operation handle according to the target operation handle, and finally, the files to be backed up in the several target operation handle queues can be packaged in parallel according to the determined file backup type, and the obtained several packaged data packets can be backed up. In this way, the parallel processing technology can be used to greatly shorten the time required for backup, while enhancing the scalability and flexibility of the system to meet the needs of large-scale data backup. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A flowchart of a file backup method disclosed in this application;

[0047] Figure 2 A timing diagram of a file backup method disclosed in this application;

[0048] Figure 3 A flowchart of another file backup method disclosed in this application;

[0049] Figure 4This is a schematic diagram of the structure of a file backup device disclosed in this application;

[0050] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] In the prior art, the efficiency of file backup is low and there is a lack of support for multi-threaded processing, which limits the performance potential in a multi-core processor environment and fails to effectively accelerate the backup process.

[0053] In order to overcome the above technical problems, the present application discloses a file backup method, device, equipment and storage medium, which can use parallel processing technology to significantly shorten the time required for backup, while enhancing the scalability and flexibility of the system to meet the needs of large-scale data backup.

[0054] See also Figure 1 As shown, an embodiment of the present invention discloses a file backup method, which is applied to a file system, comprising:

[0055] Step S11: Scan the file system to determine a number of files to be backed up in the file system.

[0056] In this embodiment, it is necessary to scan the file system through the file system traversal module in the file system, and then determine a number of files to be backed up in the file system. It should be noted that the function of the file system traversal module is to traverse all files that need to be backed up in the file system, and the file system traversal module is developed in Java language, making full use of the Java standard library to identify file types.

[0057] Step S12: traverse the plurality of files to be backed up to determine the file type corresponding to the current file to be backed up among the plurality of files to be backed up.

[0058] In this embodiment, Figure 2As shown, it is necessary to traverse several files to be backed up, and determine the attribute value of the current file to be backed up among the several files to be backed up through a first preset method. It should be noted that the first preset method is the Files.readAttributes method, and the first attribute value is the unix:nlink attribute value of the file. If the attribute value is greater than the preset attribute value threshold, it is determined that the file type of the current file to be backed up is a hard link file, that is, it is necessary to call the Files.readAttributes method to detect the unix:nlink attribute value of the file. If the attribute value exceeds 1, it is confirmed that the file type is a hard link file; in another case, if the attribute value is not greater than the preset attribute value threshold, the second preset method is used to detect whether the file type of the current file to be backed up is a soft link file, and a first return result corresponding to the second preset method is obtained. The second preset method is the Files.isSymbolicLink method, that is, if the unix:nlink If the attribute value is less than 1, the file type of the current file to be backed up needs to be detected by the Files.isSymbolicLink method. If the first return result is true, it indicates that the file type is a soft link file; if the first return result indicates that the current file to be backed up is not a soft link file, it is necessary to detect whether the file type of the current file to be backed up is a directory file through a third preset method, and obtain a second return result corresponding to the third preset method, wherein the third preset method is the Files.isDirectory method, that is, if the first return result indicates that the current file to be backed up is not a soft link file, it is necessary to detect whether the file type of the current file to be backed up is a directory file through the Files.isDirectory method. If the second return result is true, it indicates that the file type is a soft link file; finally, if the second return result indicates that the current file to be backed up is not a directory file, it can be directly determined that the file type of the current file to be backed up is a regular file. In this way, different types of files in the file system can be accurately identified.

[0059] Step S13: Generate a target operation handle corresponding to the file type, and add the current file to be backed up to a target operation handle queue corresponding to the target operation handle based on the target operation handle.

[0060] In this embodiment, it is necessary to generate a target operation handle corresponding to the file type, and to construct a target operation handle queue corresponding to the file type. After determining the target operation handle corresponding to the current file to be backed up, it is necessary to establish a corresponding relationship between the current file to be backed up and the target operation handle, so as to add the current file to be backed up to the target operation handle queue corresponding to the target operation handle through the target operation handle. For example, if the file type of the current file to be backed up is a hard link file, it is necessary to generate a hard link handle corresponding to the current file to be backed up, and then add the current file to be backed up to the hard link queue through the hard link handle.

[0061] It should be noted that since file types include hard link files, soft link files, directory files and regular files, the corresponding target operation handles include hard link handles, soft link handles, directory handles and file handles, and the target operation handle queue includes a hard link queue corresponding to the hard link handle, a soft link queue corresponding to the soft link handle, a directory queue corresponding to the directory handle and a file queue corresponding to the file handle.

[0062] It is further explained that the operation handle queue is also built based on Java, and uses LinkedBlockingQueue as the data structure to implement the queue function. Among them, the queue's put method is responsible for the queue operation. When the queue capacity is saturated, it automatically blocks subsequent queue requests to effectively prevent memory overflow. The take method performs the queue operation and blocks when the queue is idle until a new element is added. Whether it is inserting or removing an element, LinkedBlockingQueue can ensure that other elements do not need to be moved during the operation, which greatly improves the response speed and overall throughput efficiency of queue operations.

[0063] Step S14, taking the next file to be backed up as the current file to be backed up, and jumping to the step of traversing the plurality of files to be backed up to determine the file type corresponding to the current file to be backed up among the plurality of files to be backed up, until the plurality of files to be backed up are all added to the corresponding target operation handle queue.

[0064] In this embodiment, after the current file to be backed up is added to the corresponding target operation handle queue, it is necessary to use the next file to be backed up among all the traversed files to be backed up as the current file to be backed up, and then jump to the step of traversing the several files to be backed up to determine the file type corresponding to the current file to be backed up among the several files to be backed up, and repeat the step of adding the current file to be backed up to the corresponding target operation handle queue until all the files to be backed up are added to the corresponding target operation handle queue.

[0065] Step S15: Packing the to-be-backed files in the target operation handle queues in parallel according to the determined file backup type, and backing up the obtained several packed data packets.

[0066] In this embodiment, it is necessary to perform parallel packaging operations on the files to be backed up in several target operation handle queues according to the file backup type. In the first specific implementation case, if the file backup type is a full backup, it is necessary to record the start time of the first backup as a time reference for distinguishing new and old data in subsequent incremental backups. The file traversal module traverses the directory to be backed up. If the directory is accessed, the operation handle of the directory is placed in the directory operation handle queue. The directory packaging module listens to the directory operation handle queue, obtains the directory operation handle, assembles the directory data packet header, and appends the directory name to the directory data packet header, and finally outputs the directory data packet to the data packet target path; if the file is accessed, the operation handle of the file is placed in the file operation handle queue, the standard file packaging module listens to the file operation handle, submits the packaging task to the file packaging thread pool, the packaging thread assembles the file data packet header through the file operation handle, appends the file name and file data to the file data packet header, and finally outputs the file data packet header to the data packet target path; if the soft link is accessed, the operation handle of the soft link is placed in the soft link operation handle queue, the soft link packaging module listens to the soft link operation handle queue, Link operation handle queue, obtain soft link operation handle, assemble soft link data packet header, and append the target path of soft link to the back of soft link data packet header, and finally output soft link data packet to the target path of data packet; if hard link is accessed, first check whether this hard link exists in hard link cache (hard link cache is key-value structure, key is hard link inode, value is hard link target path and the number of hard links corresponding to this inode), if it exists, obtain the target path of hard link from hard link cache, and reduce the number of hard links corresponding to inode recorded in cache by one, if the result is 0, it means that all hard links corresponding to this inode have been accessed, in order to reduce cache space occupation, delete this hard link from cache, and put the hard link handle into hard link handle queue, hard link packaging module listens to hard link handle queue, assembles hard link data packet header, and appends hard link target path to the back of hard link data packet header; if it does not exist, build hard link cache, and put hard link handle into file operation handle queue, and standard file packaging module performs packaging operation. Finally, all traversed file names are recorded as the basis for subsequent incremental backups to record the files to be deleted.

[0067] In the second specific implementation, if the file backup type is incremental backup, the second backup start time is recorded first as in full backup, and the first backup start time and modification time corresponding to several files to be backed up are determined. When the file traversal module traverses the backup directory, it determines whether the modification time of the currently accessed file is greater than the first backup start time. If it is greater than the first backup start time, it means that the file has been modified after the last backup, or the file was newly created after the last backup. Otherwise, it is considered that the file has not changed after the last backup, and the incremental backup operation is stopped. Incremental backup only needs to back up changed or newly created files. Files that existed in the last backup but do not exist in this backup are considered to be files deleted after the last backup. The file names of these files are recorded. These files need to be deleted during recovery, and the second backup start time is used as the first backup start time for the next judgment.

[0068] It can be seen that, through the method of the present application, several files to be backed up in the file system can be scanned and the file types corresponding to the several files to be backed up can be determined, and then a target operation handle corresponding to the file type can be generated, and the files to be backed up can be added to the target operation handle queue corresponding to the target operation handle according to the target operation handle, and finally, the files to be backed up in the several target operation handle queues can be packaged in parallel according to the determined file backup type, and the obtained several packaged data packets can be backed up. In this way, the parallel processing technology can be used to greatly shorten the time required for backup, while enhancing the scalability and flexibility of the system to meet the needs of large-scale data backup.

[0069] See also Figure 3 As shown, an embodiment of the present invention discloses a file backup method, comprising:

[0070] Step S21: If the file backup type is full backup, the first backup start time is recorded, and the files to be backed up in the directory queue are packaged by a preset directory packaging component based on a first preset data encapsulation structure to obtain a packaged directory data packet.

[0071] In this embodiment, it is necessary to package the files to be backed up in the directory queue through a preset directory packaging component, that is, a directory packaging module, wherein the directory packaging module adopts a single-threaded strategy to ensure the stability and consistency of the processing process. The directory packaging module extracts tasks from the directory operation handle queue in an orderly manner, and then starts the packaging work of the directory data. The generated data packet follows the first preset data encapsulation structure: "Directory 1 data packet header + directory 1 name + directory 2 data packet header + directory 2 name +... + directory N data packet header + directory N name". Due to the variability of the length of the directory name, in order to eliminate the potential redundancy inside the data packet and ensure that each directory data packet can be accurately divided during the recovery phase, it is necessary to set the data packet header to a fixed length, in which the specific length of the directory name is recorded to improve data density and parsing accuracy. The data packet header occupies 21 bytes of space, and the layout is as follows: the first byte carries the version identifier of the data packet header, reserving space for future upgrades and compatibility expansion; since the Linux system sets the upper limit of the directory name length to 4096 bytes, two bytes (the second to third bytes) are required to save the directory length information, which meets the conversion requirements of bytes and bits. The next two bytes (bytes 4 to 5) are used to record the Linux directory permission information. In order to accurately match the range of user identifiers (UIDs) and group identifiers (GIDs) in the Linux system, 8 bytes of storage space are allocated to each user identifier (UID) and group identifier (GID) (bytes 6 to 13 and bytes 14 to 21 for UID), ensuring the integrity and accuracy of the authentication information. This not only reduces data holes in data packets, but also lays a solid foundation for the scalability of the system.

[0072] Step S22: Pack the files to be backed up in the file queue based on the second preset data encapsulation structure by using a preset file packaging component to obtain a packaged file data packet.

[0073] In this embodiment, it is necessary to package the files to be backed up in the file queue through a preset file packaging component, that is, a standard file packaging module, wherein the standard file packaging module adopts a multi-threaded architecture design to enhance the processing speed of file packaging. The standard file packaging module extracts tasks from the file queue and delegates the packaging job to the standby thread in the thread pool, thereby packaging and outputting the file data in parallel. The generated data packet follows the second preset data encapsulation structure: "File 1 data packet header + file 1 name + file 1 actual data + file 2 data packet header + file 2 name + file 2 actual data +... + file N data packet header + file N name + file N actual data". Similar to the directory data packet, in order to ensure accurate distinction and parsing of the data packet content of each file, a file data packet header of uniform length is used to record the file name and content length in detail, thereby ensuring the integrity and identifiability of the data. The file data packet header occupies 71 bytes, and its layout is as follows: the first byte is dedicated to marking the version number of the data packet header, which reserves space for future iterations and upgrades; the two bytes that follow (bytes 2 to 3) are used to encode the length of the file name, ensuring flexible support for file names of different lengths; bytes 4 to 5 record the file permission information in the Linux system style to ensure that the permissions remain unchanged when the file is restored; and the user identifier (UID) and group identifier (GID) occupy bytes 6 to 13 and 14 to 21 respectively. It is particularly worth mentioning that bytes 22 to 71 are used to store records of the length of the file content. With these 50 bytes, it is theoretically possible to represent 2 to the power of 50 bytes, that is, 1 PiB file size.

[0074] Step S23: Pack the files to be backed up in the soft link queue based on the third preset data encapsulation structure through the preset soft link packaging component to obtain a packaged soft link data packet.

[0075] In this embodiment, it is necessary to package the files to be backed up in the soft link queue through a preset soft link packaging component, that is, a soft link packaging module, wherein the soft link packaging module adopts a single-threaded design, and after obtaining the soft link operation handle from the soft link queue, it starts to package and output the soft link data packet. The format of the data packet conforms to the third preset data encapsulation structure, that is, "soft link 1 data packet header + soft link 1 name + soft link 1 target path + soft link 2 data packet header + soft link 2 name + soft link 2 target path + ... + soft link N data packet header + soft link N name + soft link N target path", the length of the soft link data packet header is 21 bytes, the first byte marks the data packet header version; the 2nd to 3rd bytes are allocated to the soft link name length field to adapt to link names of different lengths; since the soft link itself does not have a modifiable permission attribute, the permission record is omitted, and the user ID (UID) is directly recorded from the 4th to the 11th bytes, and the following 12th to the 19th bytes are used to record the group ID (GID); finally, the 20th to the 21st bytes are used to indicate the length of the soft link target path to ensure the complete preservation of the target location information.

[0076] Step S24: Pack the to-be-backed-up files in the hard link queue based on the third preset data encapsulation structure by using a preset hard link packing component to obtain a packed hard link data packet.

[0077] In this embodiment, it is necessary to package the files to be backed up in the hard link queue through a preset hard link packaging component, that is, a hard link packaging module, wherein the hard link packaging module adopts a single-thread design, and after obtaining the hard link operation handle from the hard link operation handle queue, it starts to package and output the hard link data packet. The format of the data packet conforms to the third preset data encapsulation structure, that is, "hard link 1 data packet header + hard link 1 name + hard link 1 target path + hard link 2 data packet header + hard link 2 name + hard link 2 target path + ... + hard link N data packet header + hard link N name + hard link N target path", and the header part of the hard link is optimized to 5 bytes: the first byte is fixed to identify the version information of the header; the second to third bytes immediately following it are allocated to the encoding of the hard link name length, which is adapted to link names of different lengths; since the hard link automatically inherits the permissions, user ID (UID) and group ID (GID) of the target file, redundant records are avoided, so the length of the hard link target path is directly specified by the fourth to fifth bytes.

[0078] It can be seen that, through the method of the present application, different data encapsulation structures can be set for different target operation handle queues, and by designing a diversified packaging processing module, the parallel processing capabilities of modern multi-core processors can be fully utilized, especially the multi-threaded architecture is adopted in the standard file packaging module, which greatly accelerates the packaging and output process of file data, makes hardware resources more fully utilized, and improves the overall processing capacity of the system. The designed data packet header structure not only ensures the integrity and fast parsing of the data, but also reduces the redundant information in the data packet through the fixed-length header design, improves the efficiency of data transmission and storage, and also provides guarantees for accurate matching and permission restoration during the recovery process.

[0079] See also Figure 4 As shown, an embodiment of the present invention discloses a file backup device, which is applied to a file system, including:

[0080] A file scanning module 11, used for scanning a file system to determine a number of files to be backed up in the file system;

[0081] A file type determination module 12, configured to traverse the plurality of files to be backed up to determine a file type corresponding to a current file to be backed up among the plurality of files to be backed up;

[0082] A file adding module 13, configured to generate a target operation handle corresponding to the file type, so as to add the current file to be backed up to a target operation handle queue corresponding to the target operation handle based on the target operation handle;

[0083] A step jump module 14 is used to take the next file to be backed up as the current file to be backed up, and jump to the step of traversing the plurality of files to be backed up to determine the file type corresponding to the current file to be backed up among the plurality of files to be backed up, until the plurality of files to be backed up are all added to the corresponding target operation handle queue;

[0084] The data backup module 15 is used to package the to-be-backed files in a plurality of target operation handle queues in parallel according to the determined file backup types, and to back up the obtained plurality of packaged data packets.

[0085] In the present application, it is first necessary to scan the file system to determine a number of files to be backed up in the file system; traverse the number of files to be backed up to determine the file type corresponding to the current file to be backed up among the number of files to be backed up; generate a target operation handle corresponding to the file type, and add the current file to be backed up to the target operation handle queue corresponding to the target operation handle based on the target operation handle; take the next file to be backed up as the current file to be backed up, and jump to the step of traversing the number of files to be backed up to determine the file type corresponding to the current file to be backed up among the number of files to be backed up, until the number of files to be backed up are all added to the corresponding target operation handle queue; package the files to be backed up in the target operation handle queue in parallel according to the determined file backup type, and back up the obtained number of packaged data packets. It can be seen that, through the method of the present application, several files to be backed up in the file system can be scanned and the file types corresponding to the several files to be backed up can be determined, and then a target operation handle corresponding to the file type can be generated, and the files to be backed up can be added to the target operation handle queue corresponding to the target operation handle according to the target operation handle, and finally, the files to be backed up in the several target operation handle queues can be packaged in parallel according to the determined file backup type, and the obtained several packaged data packets can be backed up. In this way, the parallel processing technology can be used to greatly shorten the time required for backup, while enhancing the scalability and flexibility of the system to meet the needs of large-scale data backup.

[0086] In some embodiments, the file type determination module 12 may specifically include:

[0087] An attribute value determining unit, configured to traverse the plurality of files to be backed up, and determine an attribute value of a current file to be backed up among the plurality of files to be backed up by a first preset method;

[0088] A first file type determination unit, configured to determine that the file type of the current file to be backed up is a hard link file if the attribute value is greater than a preset attribute value threshold;

[0089] A second file type determination unit, configured to detect whether the file type of the current file to be backed up is a soft link file by a second preset method if the attribute value is not greater than the preset attribute value threshold, and obtain a first return result corresponding to the second preset method;

[0090] A third file type judgment unit is used to detect whether the file type of the current file to be backed up is a directory file by a third preset method if the first returned result indicates that the current file to be backed up is not a soft link file, and obtain a second returned result corresponding to the third preset method;

[0091] The fourth file type determination unit is configured to determine that the file type of the current file to be backed up is a regular file if the second returned result indicates that the current file to be backed up is not a directory file.

[0092] In some embodiments, the file adding module 13 may specifically include:

[0093] A file adding unit, used for generating a target operation handle corresponding to the file type, and establishing a corresponding relationship between the current file to be backed up and the target operation handle, so as to add the current file to be backed up to a target operation handle queue corresponding to the target operation handle through the target operation handle;

[0094] A handle destroying unit, used for destroying the target operation handle if the previous file to be backed up has been added to the target operation handle queue;

[0095] Among them, the target operation handle includes a hard link handle, a soft link handle, a directory handle and a file handle, and the target operation handle queue includes a hard link queue corresponding to the hard link handle, a soft link queue corresponding to the soft link handle, a directory queue corresponding to the directory handle and a file queue corresponding to the file handle.

[0096] In some embodiments, the data backup module 15 may specifically include:

[0097] The data backup submodule is used to determine the file backup type according to the received backup operation instruction, and to package several files to be backed up in the target operation handle queue in parallel according to the file backup type through several preset packaging components corresponding to the file type, and to back up the obtained packaged data packets.

[0098] In some embodiments, the data backup submodule may specifically include:

[0099] A first data packaging unit is used to record a first backup start time if the file backup type is a full backup, and to package the files to be backed up in the directory queue based on a first preset data encapsulation structure through a preset directory packaging component to obtain a packaged directory data packet;

[0100] A second data packaging unit, configured to package the files to be backed up in the file queue based on a second preset data encapsulation structure through a preset file packaging component to obtain a packaged file data packet;

[0101] A third data packaging unit is used to package the to-be-backed-up files in the soft link queue based on a third preset data encapsulation structure through a preset soft link packaging component to obtain a packaged soft link data packet;

[0102] The fourth data packaging unit is used to package the files to be backed up in the hard link queue based on the third preset data encapsulation structure through a preset hard link packaging component to obtain a packaged hard link data packet.

[0103] In some embodiments, the data backup module 15 may specifically include:

[0104] A time recording unit, for recording a second backup start time if the file backup type is an incremental backup, and determining the first backup start time and modification time corresponding to the plurality of files to be backed up;

[0105] An incremental backup stopping unit, configured to stop the incremental backup operation if the plurality of files to be backed up do not have a modification time, or the modification time is not greater than the first backup start time;

[0106] The incremental backup unit is used to perform incremental backup on the plurality of files to be backed up if the modification time is greater than the first backup start time, and use the second backup start time as the first backup start time for next determination.

[0107] In some embodiments, the file backup device may further include:

[0108] The data recovery unit is used to read the plurality of packaged data packets and parse the plurality of packaged data packets if a data recovery instruction is received, so as to perform a data recovery operation through the plurality of to-be-recovered files corresponding to the plurality of packaged data packets.

[0109] Furthermore, the present application also discloses an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0110] Figure 5 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the file backup method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0111] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0112] In addition, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0113] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the file backup method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0114] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed file backup method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0115] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0116] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0117] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0118] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0119] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A file backup method, characterized in that: Applies to file systems, including: Scanning a file system to determine a number of files to be backed up in the file system; Traversing the plurality of files to be backed up to determine a file type corresponding to a current file to be backed up among the plurality of files to be backed up; Generate a target operation handle corresponding to the file type, and add the current file to be backed up to a target operation handle queue corresponding to the target operation handle based on the target operation handle; The next file to be backed up is used as the current file to be backed up, and the process jumps to the step of traversing the plurality of files to be backed up to determine the file type corresponding to the current file to be backed up among the plurality of files to be backed up, until the plurality of files to be backed up are all added to the corresponding target operation handle queue; The files to be backed up in a plurality of target operation handle queues are packaged in parallel according to the determined file backup type, and the obtained plurality of packaged data packets are backed up.

2. The file backup method according to claim 1, characterized in that: The traversing the plurality of files to be backed up to determine the file type corresponding to the current file to be backed up among the plurality of files to be backed up comprises: Traversing the plurality of files to be backed up, and determining the attribute value of the current file to be backed up among the plurality of files to be backed up by a first preset method; If the attribute value is greater than a preset attribute value threshold, determining that the file type of the current file to be backed up is a hard link file; If the attribute value is not greater than the preset attribute value threshold, detecting whether the file type of the current file to be backed up is a soft link file by a second preset method, and obtaining a first return result corresponding to the second preset method; If the first returned result indicates that the current file to be backed up is not a soft link file, detecting whether the file type of the current file to be backed up is a directory file by a third preset method, and obtaining a second returned result corresponding to the third preset method; If the second returned result indicates that the current file to be backed up is not a directory file, then the file type of the current file to be backed up is determined to be a regular file.

3. The file backup method according to claim 1, characterized in that: The step of generating a target operation handle corresponding to the file type, and adding the current file to be backed up to a target operation handle queue corresponding to the target operation handle based on the target operation handle, comprises: Generate a target operation handle corresponding to the file type, and establish a corresponding relationship between the current file to be backed up and the target operation handle, so as to add the current file to be backed up to a target operation handle queue corresponding to the target operation handle through the target operation handle; If the file to be backed up has been added to the target operation handle queue, destroying the target operation handle; Among them, the target operation handle includes a hard link handle, a soft link handle, a directory handle and a file handle, and the target operation handle queue includes a hard link queue corresponding to the hard link handle, a soft link queue corresponding to the soft link handle, a directory queue corresponding to the directory handle and a file queue corresponding to the file handle.

4. The file backup method according to claim 3, characterized in that: The step of packaging the files to be backed up in the target operation handle queues in parallel according to the determined file backup types, and backing up the obtained packaged data packets, includes: The file backup type is determined according to the received backup operation instruction, and the to-be-backed files in the target operation handle queues are packaged in parallel according to the file backup type through several preset packaging components corresponding to the file type, and the obtained packaged data packets are backed up.

5. The file backup method according to claim 4, characterized in that: The method of packaging the files to be backed up in the target operation handle queues in parallel according to the file backup type by using the preset packaging components corresponding to the file type, and backing up the obtained packaged data packets includes: If the file backup type is a full backup, the first backup start time is recorded, and the to-be-backed-up files in the directory queue are packaged by a preset directory packaging component based on a first preset data encapsulation structure to obtain a packaged directory data packet; and, packaging the to-be-backed-up files in the file queue based on a second preset data encapsulation structure by a preset file packaging component to obtain a packaged file data packet; and, packaging the to-be-backed-up files in the soft link queue based on a third preset data encapsulation structure by a preset soft link packaging component to obtain a packaged soft link data packet; and, packaging the to-be-backed-up files in the hard link queue based on the third preset data encapsulation structure by a preset hard link packaging component to obtain a packaged hard link data packet.

6. The file backup method according to claim 5, characterized in that: The step of packaging the files to be backed up in the target operation handle queues in parallel according to the determined file backup types, and backing up the obtained packaged data packets, includes: If the file backup type is incremental backup, then record the second backup start time, and determine the first backup start time and modification time corresponding to the plurality of files to be backed up; If the plurality of files to be backed up do not have modification time, or the modification time is not greater than the first backup start time, then the incremental backup operation is stopped; If the modification time is greater than the first backup start time, incremental backup is performed on the plurality of files to be backed up, and the second backup start time is used as the first backup start time for next determination.

7. The file backup method according to any one of claims 1 to 6, characterized in that: After the method of packaging the files to be backed up in the plurality of target operation handle queues in parallel according to the determined file backup types and backing up the plurality of packaged data packets obtained, the method further includes: If a data recovery instruction is received, the plurality of packaged data packets are read and parsed, so as to perform a data recovery operation through the plurality of files to be recovered corresponding to the plurality of packaged data packets.

8. A file backup device, characterized in that: Applies to file systems, including: A file scanning module, used for scanning the file system to determine a number of files to be backed up in the file system; A file type determination module, used for traversing the plurality of files to be backed up to determine a file type corresponding to a current file to be backed up among the plurality of files to be backed up; A file adding module, used for generating a target operation handle corresponding to the file type, so as to add the current file to be backed up to a target operation handle queue corresponding to the target operation handle based on the target operation handle; a step jump module, used for taking the next file to be backed up as the current file to be backed up, and jumping to the step of traversing the plurality of files to be backed up to determine the file type corresponding to the current file to be backed up among the plurality of files to be backed up, until the plurality of files to be backed up are all added to the corresponding target operation handle queue; The data backup module is used to package several files to be backed up in the target operation handle queue in parallel according to the determined file backup type, and to back up the obtained several packaged data packets.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the file backup method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the file backup method according to any one of claims 1 to 7 is implemented.