File system data migration method, system and device based on FUSE and medium

Abstract

The invention relates to a file system data migration method, system and device based on FUSE and a medium, and belongs to the technical field of data migration protection. The system comprises a data protection file system module, a FUSE file system module, a detection and feedback module and a system configuration and maintenance management module. The data protection file system module comprises a metadata sub-module and a data snapshot sub-module, the metadata sub-module is used for data mapping and related metadata management, and the data snapshot sub-module is used for data backup, unique ID management and verification management; the FUSE file system module is used for performing mixed storage, compression and transmission on data; the detection and feedback module is used for recording file operation and counting data traffic; the system configuration and maintenance management module is used for managing system configuration and daily maintenance tasks; the file system is realized under the condition of not invading the kernel, the switching between the user mode and the kernel mode is reduced, the data migration efficiency of the file system is improved, and the real-time backup protection of the data is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data migration protection, and in particular relates to a FUSE-based file system data migration method, system, device and medium. Background Art

[0002] Data protection is the process of copying all or part of the data set from the hard disk or array of the application host to other storage media to prevent data loss due to system operation errors or system failures. The current mainstream data protection methods include: tape drive backup of related data, backup database backup of main database data, local network storage backup data, and off-site storage backup data.

[0003] At present, in the field of data migration, various backup software are widely used to achieve redundant data backup. However, as an independent application, the simple file copy method of third-party backup software is not suitable for applications such as databases, because when the opened file has restrictions that cannot be copied, the database must be closed first if it is to be backed up, which is obviously unrealistic. In addition, third-party backup software has the problem of insufficient real-time performance during data transmission, that is, data is first written to the host and then synchronized to the backup machine, which may lead to untimely update of the backup machine data. If a failure occurs at this time, the data of the backup machine may not be complete and continuous. At the same time, the use of third-party backup software may also cause a decline in server performance. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention proposes a file system data migration method, system, device and medium based on FUSE.

[0005] The purpose of the present invention can be achieved through the following technical solutions: A FUSE-based file system data migration system includes a data protection file system module, a FUSE file system module, a detection and feedback module, and a system configuration and maintenance management module; The data protection file system module includes a metadata submodule and a data snapshot submodule, wherein the metadata submodule is used for data mapping and related metadata management, and the data snapshot submodule is used for data backup, unique ID management, and verification management; The FUSE file system module is used to store, compress and transmit the acquired data through the file system kernel state interface and the file system user state interface, and includes a file system kernel state interface submodule and a file system user state interface submodule; The detection and feedback module is used to record file operations and statistical data traffic, including a file operation recording submodule and a data traffic statistics submodule; The system configuration and maintenance management module is used to manage system configuration and daily maintenance tasks, and includes a file system mounting and unmounting submodule, a remote storage fault short-circuit switch submodule, and a log management submodule.

[0006] Preferably, the data mapping includes local Super block data mapping, Dentry data mapping, and Inode data mapping, and the related metadata management includes but is not limited to namespace management, storage path management, file attribute management, quota management, and file verification management.

[0007] Preferably, the local Super block data mapping is used to manage the global information of the file system, and the global information includes but is not limited to the block size and the number of inodes; The Dentry data mapping is used to maintain the correspondence between the file name and the inode so as to quickly locate the file; The Inode data mapping is used to store metadata information of the file, and the metadata information includes but is not limited to the file size and creation time.

[0008] Preferably, the data backup includes snapshot creation, idempotence guarantee, and consistency guarantee; The unique ID management assigns a unique identifier to each file operation to prevent duplicate writing of data in related backup operations; The verification management includes data verification and interface verification. The data verification is used to perform data verification after creating a snapshot to ensure that the data in the snapshot is consistent with the original data. The interface verification is used to verify the relevant interfaces to ensure that when calling the backup operation, the parameters passed in and the execution results are correct and consistent.

[0009] Preferably, the snapshot creation is used to create a snapshot of the data for each operation, and the snapshot contains the state of the data at a specific point in time to ensure that the previous state can be restored in the event of data loss or damage. The idempotence guarantee is used to design the snapshot operation to be idempotent to ensure that the result remains the same after repeated execution of the operation. The consistency guarantee is used to ensure the consistency of the data during the snapshot operation.

[0010] Preferably, the file system kernel state interface submodule processes data through the kernel state interface, directly intercepts data at the VFS layer, implements kernel state data reading and writing, and maintains kernel state data cache; The file system user state interface submodule implements file system operations in the user state, interacts with the kernel state interface through FUSE, and implements data reading and writing in the user state.

[0011] Preferably, the file system mounting and unmounting submodule is used to provide the mounting and unmounting functions of the data protection file system; the remote storage fault short-circuit switch submodule is used to monitor the remote storage status, and when a remote storage fault occurs, a short-circuit switch is provided to quickly switch the storage path to the backup storage; The log management submodule includes log level and switch configuration and log data rolling deletion; the log data rolling deletion is performed according to file time, size, etc. The log includes system fault log records and data operation log records, which are used to record log information during system operation.

[0012] A FUSE-based file system data migration method, applied to the above-mentioned FUSE-based file system data migration system, comprises: When the application is mounted on FUSE and executes a system call, the system call operation is routed to the fuse driver through VFS. The FUSE includes a kernel module and a fuse daemon. The fuse driver creates a fuse request structure and saves the request in the request queue. The process executing the operation will be blocked. The fuse daemon reads the fuserequest of the kernel queue in the kernel module through the / dev / fuse block device, and submits the operation to the underlying file system, wherein the underlying file system includes EXT4 and F2FS; When the underlying file system completes processing the request, the fuse daemon writes a reply back to the / dev / fuse block device, and the fuse driver marks the request as completed and wakes up the user process.

[0013] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned FUSE-based file system data migration system when executing the computer program.

[0014] A storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute the above-mentioned FUSE-based file system data migration system when executed by a computer processor.

[0015] The beneficial effects of the present invention are: (1) By introducing FUSE, non-intrusive kernel interaction is achieved, the file system is implemented without intruding the kernel, and file system data is migrated in user space. The file system is directly used for incremental backup at the bottom layer without modifying the kernel code. This reduces the switching between user mode and kernel mode when reading and transmitting data, effectively improving the efficiency of data migration.

[0016] (2) Snapshot creation ensures that the previous state can be restored in the event of data loss or corruption. Idempotence ensures that the results obtained from repeated executions are the same. Consistency guarantees ensure data consistency during snapshot operations, thus solving the problem of untimely data updates.

[0017] (3) By designing a cache area, data is sent in batches only after a certain amount of data has accumulated or a certain period of time has passed, thus avoiding unnecessary memory allocation and copying, meeting the need to back up local data to a remote end, reducing the system call overhead caused by frequent sending of multiple groups of small data volumes, and greatly improving the speed of sending data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0019] Figure 1 The figure is a schematic diagram of a FUSE-based file system data migration system of the present invention. DETAILED DESCRIPTION

[0020] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0021] See also Figure 1 , a FUSE-based file system data migration system, including a data protection file system module, a FUSE file system module, a detection and feedback module, and a system configuration and maintenance management module; The data protection file system module includes a metadata submodule and a data snapshot submodule, wherein the metadata submodule is used for data mapping and related metadata management, and the data snapshot submodule is used for data backup, unique ID management, and verification management; The FUSE file system module is used to store, compress and transmit the acquired data through the file system kernel state interface and the file system user state interface, and includes a file system kernel state interface submodule and a file system user state interface submodule; The detection and feedback module is used to record file operations and statistical data traffic, including a file operation recording submodule and a data traffic statistics submodule; The system configuration and maintenance management module is used to manage system configuration and daily maintenance tasks, and includes a file system mounting and unmounting submodule, a remote storage fault short-circuit switch submodule, and a log management submodule.

[0022] In this embodiment, the data mapping includes local Super block data mapping, Dentry data mapping, Inode data mapping, and the related metadata management includes but is not limited to namespace management, storage path management, file attribute management, quota management, and file verification management; The local Super block data mapping is used to manage the global information of the file system, and the global information includes but is not limited to the block size and the number of inodes; The Dentry data mapping is used to maintain the correspondence between the file name and the inode so as to quickly locate the file; The Inode data mapping is used to store metadata information of the file, including but not limited to file size and creation time; It should be noted that local Super block, Dentry, and Inode are all file organization forms of linux vfs.

[0023] In this embodiment, the data backup includes snapshot creation, idempotency guarantee, and consistency guarantee. The snapshot creation is used to create a snapshot of the data for each operation. The snapshot contains the state of the data at a specific time point, ensuring that the previous state can be restored when data is lost or damaged. The idempotency guarantee is used to design the snapshot operation to be idempotent, ensuring that no matter how many times the operation is performed, the result is the same, and no side effects will be generated due to repeated execution. The consistency guarantee is used to ensure the consistency of the data during the snapshot operation, that is, the data state reflected by the snapshot is consistent, and there will be no partial update or non-update situation; The unique ID management assigns a unique identifier to each file operation to prevent duplicate writing of data in related backup operations; Specifically, a 64-bit ID is used, and the unique ID is indexed by an index. The index is organized in a tree structure, including a parent directory ID and a file ID. The backup file is indexed and searched by the index. The unique ID is used to ensure that the backup operation will not cause data to be written multiple times due to system errors or repeated requests, thereby avoiding data duplication or inconsistency.

[0024] The verification management includes data verification and interface verification. The data verification is used for data verification after creating a snapshot to ensure that the data in the snapshot is consistent with the original data and has not been damaged or tampered with. The interface verification is used for related interface verification to ensure that when calling the backup operation, the input parameters and the execution results are correct and consistent.

[0025] In this embodiment, the file system kernel state interface submodule processes data through the kernel state interface, directly intercepts data at the VFS layer, implements kernel state data reading and writing, and maintains kernel state data cache; The file system user state interface submodule implements file system operations in the user state, interacts with the kernel state interface through FUSE, and implements data reading and writing in the user state.

[0026] It should be noted that the mixed storage is to cache data in the memory to speed up the reading operation, the compression is to reduce the data transmission volume and improve the transmission efficiency, and the transmission is to transfer the data to the target storage location.

[0027] In this embodiment, the file operation recording submodule is used to record detailed information of file operations, and the file operations include but are not limited to reading, writing, and modifying.

[0028] The data traffic statistics submodule is used to count traffic information of the statistics, specifically, the reading traffic and writing traffic of the statistics.

[0029] In this embodiment, the file system mounting and unmounting submodule is used to provide the mounting and unmounting functions of the data protection file system; The remote storage fault short-circuit switch submodule is used to monitor the remote storage status and provide a short-circuit switch to quickly switch the storage path to the backup storage when a remote storage fault occurs; The log management submodule includes log level and switch configuration and log data rolling deletion; the log data rolling deletion is performed according to file time, size, etc. The log includes system fault log records and data operation log records, which are used to record log information during system operation.

[0030] A FUSE-based file system data migration method, comprising: When an application is mounted on FUSE and executes a system call, the system call operation is routed to the fuse driver through VFS. The FUSE includes a kernel module and a user space daemon. The fuse driver creates a fuse request structure and saves the request in the request queue. The process executing the operation will be blocked. The fuse daemon reads the fuserequest of the kernel queue in the kernel module through the / dev / fuse block device, and submits the operation to the underlying file system, wherein the underlying file system includes EXT4 and F2FS; When the underlying file system completes processing the request, the fuse daemon writes a reply back to the / dev / fuse block device, and the fuse driver marks the request as completed and wakes up the user process.

[0031] In this embodiment, mounting the application on FUSE includes mounting the specified fuse_path on the / dev / fuse device through the mount function.

[0032] In this embodiment, when the FUSE reads data through the / dev / fuse block device, the splice block of FUSE is used to perform memory zero copy. The splice block allows the user space to transfer data between two kernel memory buffers without copying the data to the user space.

[0033] Specifically, the data in the form of a buffer including a file descriptor is processed by the write_buf() method of the fuse daemon, thereby saving a memory application and copying.

[0034] In this embodiment, fuse_session_loop_mt is used to implement multi-threaded mode reading requests, which is more efficient in request processing than single-threaded mode.

[0035] Specifically, the fuse daemon processes file operation requests based on the libfuse library. In multi-threaded mode, the fuse daemon starts with one thread. If there are more than two requests in the kernel queue, other threads will be automatically generated. By default, a maximum of 10 threads are supported to process requests simultaneously.

[0036] In this embodiment, for the data to be protected, non-intrusive kernel interaction is implemented through FUSE, and file system data migration is performed in the user space without modifying the kernel code; Specifically, the data is intercepted at the VFS file system layer, and the obtained data is mixed, compressed and transmitted through the file system kernel state interface and the file system user state interface; It should be noted that the kernel module is registered as a fuse file system driver of the Linux virtual file system when loaded, and its main implementation code is located in the user space, so there is no need to recompile the kernel, and its own file system can be implemented without invading the kernel; Specifically, a custom file system is implemented in user space through FUSE, and an RPC interface is used to process remote calls between the client and the server, thereby realizing remote backup and recovery of data. The Fuse backup file metadata is stored in key-value form, and the file metadata key is a compressed code generated according to the parent directory and the file ID. The file metadata value includes file attribute information and file block information. The file metadata key is organized and stored through the index tree of the Fuse backup file system, and the backed-up file metadata value is obtained through the file metadata key index, and the actual file data is stored through the file block information.

[0037] In order to reduce the reading speed gap between the processor and the memory, a cache memory is usually used to increase the speed of request response and data transmission, that is, the data first enters the buffer and then is accessed and sent.

[0038] In this embodiment, in order to meet the requirement of backing up local data to a remote end and avoid the system call overhead caused by frequent sending of multiple groups of small data volumes, a cache module is designed. The cache module avoids unnecessary memory allocation and copying as much as possible by designing a buffer. The buffer is a character queue, including a sender buffer and a receiver buffer. The sender buffer appends data from the end of the queue, takes data from the head of the queue and sends it to the socket; The receiving buffer takes data from the head of the queue and receives data from the socket at the tail of the queue.

[0039] Specifically, the cache module stores the data in the cache area before sending the data, and sends the data in batches when the data accumulates to a certain amount or reaches a certain time, which greatly improves the speed of sending data.

[0040] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0041] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0042] The program code included on the computer readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages-such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0043] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A file system data migration system based on FUSE, characterized in that: Including data protection file system module, FUSE file system module, detection and feedback module, system configuration and maintenance management module; The data protection file system module includes a metadata submodule and a data snapshot submodule, wherein the metadata submodule is used for data mapping and related metadata management, and the data snapshot submodule is used for data backup, unique ID management, and verification management; The FUSE file system module is used to store, compress and transmit the acquired data through the file system kernel state interface and the file system user state interface, and includes a file system kernel state interface submodule and a file system user state interface submodule; The detection and feedback module is used to record file operations and statistical data traffic, including a file operation recording submodule and a data traffic statistics submodule; The system configuration and maintenance management module is used to manage system configuration and daily maintenance tasks, and includes a file system mounting and unmounting submodule, a remote storage fault short-circuit switch submodule, and a log management submodule.

2. The FUSE-based file system data migration system according to claim 1, characterized in that: The data mapping includes local Super block data mapping, Dentry data mapping, and Inode data mapping, and the related metadata management includes but is not limited to namespace management, storage path management, file attribute management, quota management, and file verification management.

3. The FUSE-based file system data migration system according to claim 2, characterized in that: The local Super block data mapping is used to manage the global information of the file system, and the global information includes but is not limited to the block size and the number of inodes; The Dentry data mapping is used to maintain the correspondence between the file name and the inode so as to quickly locate the file; The Inode data mapping is used to store metadata information of the file, and the metadata information includes but is not limited to the file size and creation time.

4. The FUSE-based file system data migration system according to claim 1, characterized in that: The data backup includes snapshot creation, idempotence guarantee, and consistency guarantee; The unique ID management assigns a unique identifier to each file operation to prevent duplicate writing of data in related backup operations; The verification management includes data verification and interface verification. The data verification is used to perform data verification after creating a snapshot to ensure that the data in the snapshot is consistent with the original data. The interface verification is used to verify the relevant interfaces to ensure that when calling the backup operation, the parameters passed in and the execution results are correct and consistent.

5. The FUSE-based file system data migration system according to claim 4, characterized in that: The snapshot creation is used to create a snapshot of the data for each operation. The snapshot contains the state of the data at a specific point in time to ensure that the previous state can be restored in the event of data loss or damage. The idempotence guarantee is used to design the snapshot operation to be idempotent to ensure that the result remains the same after repeated execution of the operation. The consistency guarantee is used to ensure the consistency of the data during the snapshot operation.

6. The FUSE-based file system data migration system according to claim 1, characterized in that: The file system kernel state interface submodule processes data through the kernel state interface, directly intercepts data at the VFS layer, implements kernel state data reading and writing, and maintains kernel state data cache; The file system user state interface submodule implements file system operations in the user state, interacts with the kernel state interface through FUSE, and implements data reading and writing in the user state.

7. The FUSE-based file system data migration system according to claim 1, characterized in that: The file system mounting and unmounting submodule is used to provide the mounting and unmounting functions of the data protection file system; the remote storage fault short-circuit switch submodule is used to monitor the remote storage status and provide a short-circuit switch to quickly switch the storage path to the backup storage when a remote storage fault occurs; The log management submodule includes log level and switch configuration and log data rolling deletion; the log data rolling deletion is performed according to file time, size, etc. The log includes system fault log records and data operation log records, which are used to record log information during system operation.

8. A file system data migration method based on FUSE, characterized in that: The FUSE-based file system data migration system is applied to any one of claims 1 to 7, comprising: When the application is mounted on FUSE and executes a system call, the system call operation is routed to the fuse driver through VFS. The FUSE includes a kernel module and a fuse daemon. The fuse driver creates a fuse request structure and saves the request in the request queue. The process executing the operation will be blocked. The fuse daemon reads the fuserequest of the kernel queue in the kernel module through the / dev / fuse block device, and submits the operation to the underlying file system, wherein the underlying file system includes EXT4 and F2FS; When the underlying file system completes processing the request, the fuse daemon writes a reply back to the / dev / fuse block device, and the fuse driver marks the request as completed and wakes up the user process.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the FUSE-based file system data migration system is implemented as described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the FUSE-based file system data migration system as described in any one of claims 1-7 when executed by a computer processor.

Images