Method for backing up metadata of file on HDD, and metadata backup server

By monitoring the closing event of HDD disk write files and using the operating system interface to obtain metadata, efficient file metadata backup on HDD disk is achieved, solving the problem of high disk IO consumption in the existing technology, and improving data recovery Speed ​​and system performance.

WO2024230746A1PCT designated stage expired Publication Date: 2024-11-14CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PTE LTD

Patent Information

Application Number
PCT/CN2024/091765
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-08
Filing Date
2024-05-08
Publication Date
2024-11-14

Smart Images

  • Figure CN2024091765_14112024_PF_FP_ABST
    Figure CN2024091765_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of cloud computing, and discloses a method for backing up metadata of a file on an HDD, and a metadata backup server. The method comprises: acquiring directory information of data written by a data service on the HDD; monitoring a written file closing event of the directory information, and acquiring closed file information returned according to the written file closing event; and determining a closed file according to the closed file information, acquiring metadata of the closed file, and backing up the metadata to a metadata backup warehouse. According to the present invention, metadata of a closed file can be obtained when the metadata is still in a kernel memory as much as possible, and backup of the metadata to a metadata backup warehouse can be achieved while an extra disk IO operation is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Method for backing up file metadata on HDD disk and metadata backup server

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 8, 2023, with application number 202310530686.4 and application name “Method for backing up file metadata on HDD disk and metadata backup server”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing, and more specifically, to a method for backing up file metadata on an HDD disk and a metadata backup server. Background Art

[0003] This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0004] For storage systems, ensuring data loss and data integrity is an eternal pursuit. However, production systems still face the need to recover data after deletion for various reasons, such as accidental file deletion due to storage system software issues, accidental file deletion caused by maintenance operations, file system corruption, and user error. To ensure data loss prevention and meet user business needs, storage systems must recover these accidentally deleted files. For SSDs (Solid State Disks or Solid State Drives), a full disk scan is the ultimate safeguard. However, for HDDs (Hard Disk Drives), whose storage space is increasing but whose I / O (Input / Output) capacity remains relatively stable, a full disk scan can take weeks, severely impacting the timeliness of storage service recovery and potentially causing devastating data loss. To accelerate data recovery, backing up the metadata of files on HDDs is particularly important. However, existing backup solutions consume a significant amount of disk I / O capacity, which is often unacceptable for HDDs, which have limited performance.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method for backing up metadata of files on an HDD disk and a metadata backup server, which can complete the backup of metadata of files on an HDD disk while avoiding additional disk IO operations.

[0007] According to one aspect of the present application, a method for backing up file metadata on an HDD disk is also provided, including: obtaining directory information of data written by a data service on the HDD disk; monitoring a write file closing event of the directory information, and obtaining closed file information returned according to the write file closing event; determining a closed file based on the closed file information, obtaining metadata of the closed file, and backing up the metadata to a metadata backup warehouse.

[0008] According to another aspect of the present application, a metadata backup server is also provided, including: an acquisition module, an event module and a backup module: the acquisition module is used to obtain directory information of data written by the data service on the HDD disk; the event module is used to monitor the write file closing event of the directory information, and obtain the closed file information returned according to the write file closing event; the backup module is used to determine the closed file based on the closed file information, obtain the metadata of the closed file, and back up the metadata to the metadata backup warehouse.

[0009] According to another aspect of the present application, a storage system is also provided, including: the above-mentioned metadata backup server.

[0010] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, enable the processor to perform the above method.

[0011] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable the computer to execute the steps of the above method.

[0012] According to another aspect of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the above method steps are implemented.

[0013] Beneficial effects of the embodiments of the present invention:

[0014] In an embodiment of the present invention, directory information of data written on the HDD disk by the data service is obtained; a write file close event of the directory information is monitored, and close file information returned according to the write file close event is obtained; a closed file is determined according to the close file information, metadata of the closed file is obtained, and the metadata is backed up to a metadata backup repository. The present invention monitors the directory information of data written on the HDD disk by the data service in the operating system, and when a write file close event is detected, the close file information returned according to the write file close event is obtained, and the closed file is determined according to the close file information, thereby obtaining the metadata of the closed file while the metadata is still in kernel memory as much as possible, and backing up the metadata to the metadata backup repository while avoiding additional disk IO operations.

[0015] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be derived from these drawings without inventive effort.

[0017] 1 is a flow chart of a method for backing up file metadata on an HDD disk according to an embodiment of the present invention;

[0018] 2 is a schematic diagram of a conventional backup and a modified backup process according to an embodiment of the present invention;

[0019] FIG3 is a block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes embodiments of the present invention in more detail with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0021] The first existing HDD file metadata backup solution backs up metadata through full / incremental file scans. This solution periodically scans files stored on the disk, then calls stat on each file to obtain its metadata. It then uses the fiemap interface to obtain its extent information, and then persists each file's metadata / extent information. This allows for a full backup of file metadata or incremental backups of file metadata modified since the last backup. However, this method decouples the file traversal order from the disk storage method, requiring a large number of random read operations, which consumes excessive disk performance and impacts normal service.

[0022] The second existing HDD file metadata backup solution is a file system-specific metadata backup solution. The first way to implement this solution is to back up the inode table (a data structure in the file system that stores metadata information about files and directories, including file size, creation time, modification time, access permissions, etc.): For conventional file system designs, metadata is usually stored in a relatively continuous area. To reduce the number of disk IOs required for metadata backup, metadata backup can be completed by parsing the contents of the file system metadata area. For example, each inode in ext3 occupies 128 bytes of storage space, and each inode in ext4 occupies 256 bytes of space. A single 64K IO operation can obtain metadata for 512 ext3 partition files or 256 ext4 partition files, thereby saving an order of magnitude in disk IO compared to solution one.

[0023] Because HDD disks have a relatively low IOPS (Input / Output Operations Per Second) ceiling and stored data is generally discrete, random reads and writes are frequent in both Solution 1 and Solution 2 (1), making metadata backups very unfriendly to HDDs. The second method for implementing Solution 2 is journal area backup, which is accomplished by parsing the file system's journal area. Because the journal area contains a backup of modified metadata, metadata backup is accomplished by parsing the file system's specific journal format. However, the journal area often retains multiple modifications to the same file, resulting in low storage efficiency and significant space waste. Furthermore, the reserved journal space may be small, placing high demands on the backup service's journal reading time window. Furthermore, different file systems may have different journal formats, increasing the complexity of solution adaptation.

[0024] Conventional solutions that scan files or back up the file system metadata and journal areas inevitably require reading data from the HDD, which consumes disk IO capacity and, in turn, affects service performance. To reduce disk IO operations, metadata backup can also be accomplished by enhancing the kernel. For example, by using kernel modules or BPF (Berkeley Packet Filter, a highly efficient network packet processing technology), kernel functionality can be enhanced. When a file's metadata is modified, the modified file's metadata is persisted, completing the backup. However, this solution requires kernel-mode modifications, placing very high demands on development. Inadvertently introducing bugs can lead to system crashes. Furthermore, with kernel version changes and iterations, uncontrollable adaptation risks are introduced.

[0025] Based on the above background, the present application now proposes a file metadata backup method and metadata backup server on HDD disk. This method obtains the file's metadata information, extent information, etc. while the metadata of the written file is still in the memory, thereby completing the data backup and ultimately achieving almost zero disk overhead for file system metadata backup.

[0026] First, the terms involved are explained.

[0027] File system: The methods and data structures used by an operating system to store files on a device or partition, that is, the way files are organized on a storage device.

[0028] ext4 file system: Fourth-generation extended file system (ext4), a log file system under the Linux system, is the successor version of the ext3 file system.

[0029] Metadata: also known as intermediary data or relay data, is data that describes data, mainly information that describes data attributes, used to support functions such as indicating storage location, historical data, resource search, and file records.

[0030] Extent: It is a data structure in the file system that is used to manage the storage and retrieval of files on disk.

[0031] fiemap (a Linux file system call): Many modern file systems use extents instead of blocks to manage disks. An extent is a series of consecutive blocks, defined by the starting block plus the length. fiemap is a user-mode method for obtaining file storage extent information.

[0032] Backup: The foundation of disaster recovery, it refers to the process of copying all or part of the data set from the application host's hard disk or array to other storage media to prevent data loss due to system operation errors or system failures.

[0033] stat: is a Unix / Linux system call used to obtain file metadata information, including file type, size, permissions, creation time, modification time, access time, etc. By calling the stat function, you can obtain the file metadata information and store it in a structure.

[0034] An inode is a data structure in the file system that manages metadata about files and directories. Each file and directory has a corresponding inode. The inode contains information such as the file's attributes, permissions, size, creation time, modification time, and access time, but does not contain the actual file data.

[0035] Journal: A file system's logging mechanism, used to record file system operations and state changes. The journal records file system metadata, including file creation, modification, deletion, and movement operations, as well as file system state changes such as mounting, unmounting, and restoring operations.

[0036] inotify: It is a file system monitoring mechanism provided by the Linux kernel. It can monitor files and directories in the file system and notify applications in a timely manner when files or directories change.

[0037] In this embodiment, a method for backing up file metadata on an HDD disk is provided. FIG1 is a flowchart of the method for backing up file metadata on an HDD disk according to an embodiment of the present application. The method steps involved in FIG1 are described below.

[0038] Step S102: obtaining directory information of data written by the data service on the HDD disk.

[0039] In this step, the data service is responsible for writing data. For example, it writes data from various sources to a file in a directory on the HDD. The directory information is the directory where the written files are located. Different files can belong to the same directory or different directories.

[0040] It should be noted that HDD disks are one of the main forms of computer storage and are commonly used in desktop computers, laptops, servers and other devices.

[0041] In this step, after the file metadata backup on the HDD disk is started, the directory information of the data written by the data service is obtained so as to monitor the status of the written files under the directory information.

[0042] Step S104: monitoring a write-file close event of the directory information, and obtaining file close information returned according to the write-file close event.

[0043] In this step, a write file close event is generated to indicate that the write or modification to the write file has been completed and the write file has been closed. The status of each write file corresponding to the directory information is monitored. When a write file close event is detected, the operating system returns descriptive information and identification information related to the closed file, i.e., closed file information, based on the write file close event. The closed file information returned based on the write file close event is obtained.

[0044] It should be noted that the write file close event refers to the operation of closing a file after writing to it. File closing operations can include flushing buffers, releasing file resources, and ensuring data integrity. Closing a file ensures persistent data storage and releases file resources.

[0045] It should also be noted that the description information related to the closed file may include the modified content information of the closed file. The identification information related to the closed file may be used to identify and query the closed file.

[0046] In this step, when the file close event of writing directory information is detected, although the file is closed, the metadata is still in the memory of the operating system. Therefore, the closed file information is obtained so that the metadata can be obtained based on the file information later. Performing metadata query immediately at the closing time will not generate additional disk IO.

[0047] Step S106: determining a closed file according to the closed file information, obtaining metadata of the closed file, and backing up the metadata to a metadata backup repository.

[0048] In this step, after obtaining the closed file information, the closed write file, i.e., the closed file, can be determined. If the metadata of the closed file is still in the memory, the metadata of the closed file can be obtained in real time through the standard operating system interface, thereby completing the data backup.

[0049] It should be noted that the metadata of a file refers to data that describes the file's attributes, characteristics, structure, format, source, quality and other information. Common file metadata include: File name: the name of the file, used to identify the file. File type: the type of file, such as text file, image file, audio file, video file, etc. File size: the size of the file, expressed in units such as bytes, kilobytes, and megabytes. Creation time and modification time: the creation time and last modification time of the file. File path: the path of the file in the file system, used to identify the location of the file. File permissions: the access rights of the file, including read, write, execute and other permissions. File format: the format of the file, such as the encoding format of text files, the compression format of image files, etc. File version: the version number of the file, used to identify different versions of the file. File summary: the summary information of the file, such as the hash value and digital signature of the file.

[0050] It should also be noted that the metadata backup repository can be a local disk, a network storage device, or a cloud storage service. The specific metadata backup repository used can be selected based on actual needs and is not specifically limited in this embodiment of the present invention. System metadata is backed up to the metadata backup repository to facilitate recovery in the event of system failure or data loss.

[0051] In an embodiment of the present invention, directory information of data written on the HDD disk by the data service is obtained; a write file close event of the directory information is monitored, and close file information returned according to the write file close event is obtained; a closed file is determined according to the close file information, metadata of the closed file is obtained, and the metadata is backed up to a metadata backup repository. The present invention monitors the directory information of data written on the HDD disk by the data service in the operating system, and when a write file close event is detected, the close file information returned according to the write file close event is obtained, and the closed file is determined according to the close file information, thereby obtaining the metadata of the closed file while the metadata is still in kernel memory as much as possible, and backing up the metadata to the metadata backup repository while avoiding additional disk IO operations.

[0052] In one possible implementation, monitoring the write file closing event of the directory information and obtaining the closed file information returned based on the write file closing event can be performed according to the following steps: using the inotify component to monitor the write file closing event of the directory information and obtaining the closed file information returned by the inotify component based on the write file closing event; wherein, the closed file information includes a monitoring descriptor and a file name.

[0053] In an embodiment of the present invention, inotify can help applications monitor file system changes in real time, thereby improving the efficiency and reliability of applications. The inotify component is used to monitor write-file close events in directory information. When a write-file close event is detected, the inotify component can promptly obtain the file close information returned based on the write-file close event.

[0054] The closed file information includes a watch descriptor and a file name. A watch descriptor (WD) is a concept in the inotify mechanism that represents a monitored file or directory. When using the inotify mechanism, an application creates an inotify instance using the inotify_init() function and adds a watch directory using the inotify_add_watch() function, which returns a watch descriptor representing that directory. The watch descriptor and file name can be used to identify the full path of the closed file and uniquely locate the modified file.

[0055] It should be noted that inotify can be used for: File system monitoring: inotify can monitor files and directories in the file system, including operations such as file creation, modification, deletion, and movement. When a file or directory changes, inotify will promptly notify the application. Real-time notification: inotify can notify applications of file system changes in real time, avoiding the overhead and delay of polling the file system. Precise monitoring: inotify can accurately monitor files and directories in the file system, and can specify monitored file types, directory levels, file names, etc. Wide range of application scenarios: inotify can be applied to many scenarios, such as file synchronization, log monitoring, security auditing, etc.

[0056] In a possible implementation, determining the closed file according to the closed file information may be performed according to the following steps: generating a closed file path according to the monitoring descriptor and the file name; and determining the closed file according to the closed file path.

[0057] In the embodiment of the present invention, the monitoring descriptor and the file name are concatenated to obtain a closed file path, which can be used to find the closed file.

[0058] In this step, the closed file may be obtained based on the closed file path, providing support for obtaining metadata based on the closed file.

[0059] In a possible implementation, obtaining the metadata of the closed file may be performed according to the following steps: obtaining the metadata of the closed file using a fiemap interface and / or a stat system call.

[0060] In the embodiment of the present invention, both the fiemap interface and stat can be used to obtain metadata information of the file system, but their usage scenarios are slightly different.

[0061] The fiemap interface is primarily used to obtain the physical location of a file on disk, enabling operations such as file defragmentation and compression. fiemap can retrieve the location of all physical blocks in a file, including holes in sparse files. Using fiemap requires file system support.

[0062] The stat system call is primarily used to obtain basic file attributes, such as file size, creation time, modification time, and access time. Stat can also obtain a file's inode number, which can be used to determine whether two files are the same file. Using stat does not require root privileges and can be used by ordinary users.

[0063] In summary, if you need to obtain the physical location information of a file, you can use the fiemap interface; if you need to obtain the basic attribute information of a file, you can use the stat call, or you can use both methods to obtain the metadata of a closed file based on actual needs. The specific method of obtaining metadata can be determined based on actual needs, and the embodiment of the present invention does not make specific limitations on this.

[0064] Considering that the conventional backup path relies on the backup service to be able to perceive the modifications of all files and to normally persist the metadata of these modified files, but in actual operation, the events monitored by inotify may be lost, and the metadata backup service may be suddenly killed or abnormally exited at a certain moment, resulting in the metadata of the modified files during the period not being backed up normally. In this case, a separate correction backup channel is required to check for omissions and fill in the gaps. Therefore, in a possible implementation method, after backing up the metadata, the following steps can also be performed: obtain the first metadata of the first file managed by the data service; wherein the first metadata includes one or more of file name data, creation time data, Inode data and file length data; load the second metadata in the metadata backup warehouse; compare the first metadata and the second metadata, determine the incremental file list information based on the comparison result, obtain incremental metadata based on the incremental file list information, and back up the incremental metadata.

[0065] In this embodiment of the present invention, all files managed by a data service are considered first files, and the metadata of the first files in the data service is considered first metadata. The first metadata includes multiple types of metadata, including one or more of file name data, creation time data, inode data, and file length data. All metadata in the metadata backup repository is considered second metadata.

[0066] It should be noted that the above-mentioned two steps of obtaining the first metadata of the first file and loading the second metadata in the metadata backup warehouse are two independent steps. There is no particular order in which they are executed. The two can be executed simultaneously, or one can be executed first. The order in which they are executed can be determined according to actual needs. The embodiment of the present invention does not make specific limitations on this.

[0067] The first metadata and the second metadata are compared, and the obtained difference information is used as the comparison result. The incremental file list information is determined based on the comparison result. The incremental file list information includes multiple file name information. After obtaining the incremental file list information, the metadata of multiple files that need to be backed up again, i.e., incremental metadata, is obtained according to a method similar to step S106. The incremental metadata is then supplemented and backed up to the metadata backup warehouse, so that the backup metadata in the metadata backup warehouse corresponds to the metadata of the files managed by the data service, thereby avoiding errors and omissions in the backup metadata in the metadata backup warehouse and reducing the impact of various abnormal problems on the metadata backup results under normal backup processes.

[0068] In this step, by supplementing the backup incremental metadata, the regular backup can be checked for omissions and deficiencies, ensuring the integrity and accuracy of the metadata.

[0069] In one possible implementation, determining the incremental file list information based on the comparison results can be performed according to the following steps: if the first metadata includes the first target metadata, and the second metadata does not include the first target metadata, then the file name data corresponding to the first target metadata is added to the incremental file list information; if the second target metadata in the first metadata is different from any one or more of the file name data, creation time data, Inode data and file length data of the second target metadata in the second metadata, then the file name of the second target metadata in the first metadata is added to the incremental file list information.

[0070] In this step, if the first target metadata corresponding to a file only exists in the first metadata and does not exist in the second metadata, it means that the file may have been missed during the backup or the backup failed. Therefore, it is necessary to add the file name data of the above file corresponding to the first target metadata to the incremental file list information so as to supplement the metadata of the file to the metadata backup warehouse.

[0071] It should be noted that the first target metadata is a subset of the first metadata. The first target metadata is (or is not) part of a subset of the second metadata.

[0072] In addition, if the backup process proceeds normally, the file name data, creation time data, Inode data and file length data contained in the second target metadata in the first metadata are the same as those contained in the second target metadata in the second metadata. If any one or more of these data are different, the file name of the second target metadata in the first metadata is added to the incremental file list information.

[0073] It should be noted that the second target metadata is the metadata of a certain file. If the backup is normal, the second target metadata in the first metadata and the second metadata are the same. If they are different, it proves that there may be an error in the backup process. In this case, the content in the first metadata shall prevail and the metadata of the file shall be backed up again.

[0074] In this step, metadata information such as the file name, creation time, inode, and file length requires relatively little storage space and is stored contiguously on the storage medium. This storage can also be stored on high-performance storage devices, resulting in relatively low read overhead. Furthermore, the first metadata of the first file can also be retrieved in memory via a remote procedure call to the data service, without compromising HDD I / O performance.

[0075] In a possible implementation, obtaining the first metadata of the first file managed by the data service may be performed according to the following steps: obtaining the first metadata of the first file managed by the data service by remote procedure calling the data service and / or loading a checkpoint file of the data service.

[0076] In embodiments of the present invention, Remote Procedure Call (RPC) is a computer communication protocol used to perform remote calls between different computers or between different processes on the same computer. In RPC, the caller (client) sends a request message to the callee (server), which contains the name and parameters of the remote procedure to be called. After receiving the request message, the server executes the corresponding remote procedure and returns the result to the client. After receiving the result, the client continues to execute the local program. RPC can use different transmission protocols and encoding methods.

[0077] A checkpoint file is a file that saves the state of a process and is typically used for process recovery and migration. When a process needs to be suspended or migrated, its state information, including the process's memory image, register status, file descriptors, and other information, is written to the checkpoint file. This checkpoint file can be read and restored on another system, allowing the process to continue executing on that different system.

[0078] In this step, the first metadata of the first file being managed by the data service is determined by remote procedure call or checkpoint, or by using both remote procedure call and checkpoint file, thereby providing support for obtaining incremental metadata.

[0079] Referring to the schematic diagram of the regular backup and modified backup process shown in FIG2 , a possible implementation of the method is described below using a specific embodiment. In order to achieve a metadata backup service with almost no additional disk IO, metadata backup can be completed through two channels: regular backup and modified backup for missing or repairing.

[0080] 1. Regular backup:

[0081] a. After the backup service is started, obtain the directories where the data service writes data, and then register to monitor the closing events of files written to these directories through inotify_add_watch;

[0082] b. After the operating system discovers that the file write fd (file descriptor, an integer value used in the operating system to identify and access files or other input / output resources) in the monitoring directory has been closed, it returns the closed file monitoring descriptor (wd) and the closed file name to the metadata backup service through inotify_event. The metadata backup server can then assemble the closed file path based on these two pieces of information.

[0083] c. In the file system, you can use stat to obtain basic file metadata, such as file length and creation / modification time. The operating system also provides a fiemap interface, which allows you to efficiently obtain the storage location of a file on the file system from user mode. Under normal circumstances, even though the file in step 1b is closed, its metadata remains in the operating system's memory. Therefore, performing a metadata query immediately after closing the file does not incur additional disk I / O.

[0084] 2. Modify the backup:

[0085] The conventional backup path in step 1 relies on the backup service being able to detect all file modifications and persist the metadata of these modified files normally. However, in actual operation, the events monitored by inotify may be lost, and the metadata backup service may be suddenly killed or abnormally exit at some point, resulting in the metadata of modified or newly created files not being properly backed up. In this case, a separate correction backup channel is required to check for omissions and fill in the gaps. The basic process is as follows:

[0086] a. Obtain the current list of valid files and necessary metadata in the data service, such as file length, creation time, and inode number, through RPC or the data service's checkpoint file. Storing information in a checkpoint requires less storage space and is stored continuously on the storage medium, resulting in lower read overhead. Furthermore, it can be obtained in memory through RPC, without compromising HDD I / O performance.

[0087] b. Read the known backup metadata in the metadata backup warehouse and build a known file metadata backup table

[0088] c. Compare the entries in step 2a and step 2b. If all of the following conditions are met, the backed-up metadata is considered valid. Otherwise, complete the metadata backup by using a method similar to step 1c, which may consume disk I / O: i. Same file name; ii. Same creation time; iii. Same inode number; iv. Same file length.

[0089] The above metadata backup process, after obtaining sufficient information through stat / fiemap, can persist information including but not limited to file name (file path), file length, file creation time, inode number, extents list, etc., thereby completing metadata backup.

[0090] This solution allows metadata to be backed up using conventional paths with minimal additional HDD I / O, minimizing the impact on production services and improving backup service availability. Furthermore, this backup method requires no modifications to the storage service, making it more widely applicable.

[0091] This application provides a method for backing up file metadata on HDD disks. This method provides a universal file system metadata backup solution applicable to a variety of on- and off-cloud storage systems. By monitoring file write handle closure events in the data directory through the operating system interface and then implementing a corrective solution to detect and address any gaps, this method can ensure that the corresponding metadata is still in kernel memory at the time of retrieval, thereby avoiding additional disk I / O operations. Furthermore, all operations use standard interfaces supported by the operating system, eliminating the need for kernel adaptation and significantly reducing various risks.

[0092] The present application also provides a metadata backup server, including: an acquisition module, an event module and a backup module: the acquisition module is used to obtain directory information of data written by the data service on the HDD disk; the event module is used to monitor the write file closing event of the directory information, and obtain the closed file information returned according to the write file closing event; the backup module is used to determine the closed file according to the closed file information, obtain the metadata of the closed file, and back up the metadata to the metadata backup warehouse.

[0093] In an embodiment of the present application, the acquisition module is used to obtain the directory information of the data service writing data on the HDD disk; the event module is used to monitor the write file closing event of the directory information and obtain the closed file information returned according to the write file closing event; the backup module is used to determine the closed file according to the closed file information, obtain the metadata of the closed file, and back up the metadata to the metadata backup warehouse. In an embodiment of the present invention, the operating system monitors the directory information of the data service writing data on the HDD disk, and when a write file closing event is detected, the closed file information returned according to the write file closing event is obtained, and the closed file is determined according to the closed file information, so as to obtain the metadata of the closed file while the metadata is still in the kernel memory as much as possible, and to achieve the backup of the metadata to the metadata backup warehouse while avoiding additional disk IO operations.

[0094] In a possible implementation, the event module is specifically used to: utilize the inotify component to monitor the write file closing event of the directory information, and obtain the close file information returned by the inotify component according to the write file closing event; wherein, the close file information includes a monitoring descriptor and a file name.

[0095] In a possible implementation, the backup module is specifically configured to: generate a closing file path according to the monitoring descriptor and the file name; and determine a closing file according to the closing file path.

[0096] In a possible implementation, the backup module is specifically configured to obtain metadata of the closed file using a fiemap interface and / or a stat system call.

[0097] In a possible implementation, it also includes a correction module, which is used to: obtain the first metadata of the first file managed by the data service; wherein the first metadata includes one or more of file name data, creation time data, Inode data and file length data; load the second metadata in the metadata backup warehouse; compare the first metadata and the second metadata, and determine the incremental file list information according to the comparison result; obtain incremental metadata according to the incremental file list information, and back up the incremental metadata.

[0098] In one possible implementation, the correction module is specifically used to: if the first metadata includes first target metadata, and the second metadata does not include the first target metadata, then add the file name data corresponding to the first target metadata to the incremental file list information; if the second target metadata in the first metadata is different from any one or more of the file name data, creation time data, Inode data and file length data of the second target metadata in the second metadata, then add the file name of the second target metadata in the first metadata to the incremental file list information.

[0099] In a possible implementation, the correction module is specifically configured to obtain the first metadata of the first file managed by the data service by remote procedure calling the data service and / or loading a checkpoint file of the data service.

[0100] An embodiment of the present invention further provides a storage system, including: the above-mentioned metadata backup server.

[0101] An embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform a method according to an embodiment of the present invention.

[0102] An embodiment of the present invention further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0103] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to perform the method according to the embodiment of the present invention.

[0104] With reference to Figure 3, the block diagram of the electronic device 300 that can be used as the server or user of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0105] As shown in Figure 3, electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for the operation of device 300 can also be stored in RAM 303. Computing unit 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0106] Multiple components within electronic device 300 are connected to I / O interface 305, including an input unit 306, an output unit 307, a storage unit 308, and a communication unit 309. Input unit 306 can be any type of device capable of inputting information into electronic device 300. Input unit 306 can receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 307 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 308 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 309 allows electronic device 300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0107] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above. For example, in some embodiments, the aforementioned method may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. In some embodiments, the computing unit 301 may be configured to perform the aforementioned method in any other appropriate manner (e.g., by means of firmware).

[0108] The program code for implementing the method of the embodiment of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of an embodiment of the present invention, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open-ended, i.e., "including but not limited to." The term "based on" means "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; and the term "some embodiments" means "at least some embodiments." The modifications of "one" and "a plurality of" mentioned in the embodiments of the present invention are illustrative and non-restrictive. Those skilled in the art should understand that, unless the context clearly indicates otherwise, they should be understood as "one or more."

[0111] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0112] The various steps described in the method implementation scheme provided in the embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method implementation scheme may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0113] The term "embodiment" in this specification refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments are referenced to each other. In particular, for the embodiments of the device, equipment, and system, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts are referred to the partial description of the method embodiment.

[0114] The above-described embodiments merely illustrate several implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of the present invention. Therefore, the scope of the present invention shall be determined by the appended claims.

Claims

1. A method for backing up file metadata on a HDD disk, include: Obtain directory information of data written by the data service on the HDD disk; Monitor the write file closing event of the directory information, and obtain the closed file information returned according to the write file closing event; Determine a closed file according to the closed file information, obtain metadata of the closed file, and back up the metadata to a metadata backup warehouse.

2. The method according to claim 1, in, Monitoring the write file closing event of the directory information, and obtaining the closed file information returned according to the write file closing event, including: The inotify component is used to monitor the write file closing event of the directory information, and the close file information returned by the inotify component according to the write file closing event is obtained; wherein the close file information includes a monitoring descriptor and a file name.

3. The method according to claim 2, in, Determining a closed file according to the closed file information includes: Generate a closing file path according to the monitoring descriptor and the file name; Determine a closed file according to the closed file path.

4. The method according to claim 1, in, Get the metadata of the closed file, including: The metadata of the closed file is obtained by using a fiemap interface and / or a stat system call.

5. The method according to any one of claims 1 to 4, in, After backing up the metadata, it also includes: Acquire first metadata of a first file managed by the data service; wherein the first metadata includes one or more of file name data, creation time data, Inode data and file length data; Loading second metadata in the metadata backup repository; Comparing the first metadata with the second metadata, and determining incremental file list information according to the comparison result; According to the incremental file list information, incremental metadata is obtained, and the incremental metadata is backed up.

6. The method according to claim 5, in, Determine the incremental file list information based on the comparison results, including: If the first metadata includes first target metadata, and the second metadata does not include the first target metadata, then adding the file name data corresponding to the first target metadata to the incremental file list information; If the second target metadata in the first metadata is identical to the second target metadata in the second metadata If any one or more of the file name data, creation time data, Inode data and file length data in the first metadata are different, the file name of the second target metadata in the first metadata is added to the incremental file list information.

7. The method according to claim 5, in, Acquiring first metadata of a first file managed by the data service includes: The first metadata of the first file managed by the data service is obtained by remote procedure calling the data service and / or loading a checkpoint file of the data service.

8. A metadata backup server, include: Get module, event module and backup module: The acquisition module is used to acquire directory information of data written by the data service on the HDD disk; The event module is used to monitor the write file closing event of the directory information, and obtain the closed file information returned according to the write file closing event; The backup module is used to determine a closed file according to the closed file information, obtain metadata of the closed file, and back up the metadata to a metadata backup repository.

9. A storage system, include: The metadata backup server of claim 8.

10. An electronic device, include: processor; as well as Memory for storing programs, The program comprises instructions, which, when executed by the processor, cause the processor to perform the method steps according to any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to make the computer execute the method steps according to any one of claims 1-7.

12. A computer program product, in, The computer program product comprises a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Renaming method and renaming system for file synchronization among multiple devices

    CN103198100A

  • Continuous data protection method, device and equipment and readable storage medium

    CN111858164A

  • File sharing method of cross-end equipment and related products

    CN114356872A

  • Data processing method and storage device

    CN115858237A

  • Metadata backup method and metadata backup server for files on HDD (hard disk drive) disk

    CN117076191A

Cited By

  • Backup data recovery method and device based on FUSE system

    CN119988103A

  • Data backup device, method and equipment and storage medium

    CN120256205A

  • Database void data recovery method and device and storage medium

    CN121233549A