A backup method and recovery method for a large number of small data files
A file backup and small data technology, applied in the computer field, can solve the problems of long recovery time, unfavorable LOSF of VFS system call mechanism, inability to effectively improve the backup speed, etc., to achieve the effect of improving the speed of backup
Active Publication Date: 2022-01-07
南京云信达科技有限公司
View PDF0 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
As mentioned above, most of the disk file system designs focus on large data files, including metadata management, data layout, and I / O access process. In addition, the VFS system call mechanism is also very unfavorable to LOSF (LOSF is a list of current system tools to open files), these software-level mechanisms and implementations exacerbate the performance problems of LOSF
[0004] Most file backup software chooses to compress and transmit the source data in order to reduce the backup time, but for small data files, this method cannot effectively improve the backup speed, and at the same time, the source data needs to be decompressed during recovery, resulting in too long recovery time , unable to restore the user's production environment in time
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0023] The present invention will be further explained below in conjunction with the accompanying drawings.
[0024] The experimental hardware architecture provided in this embodiment is: storage terminal: x86 architecture, os: centos7.4, hard disk: 10T. Client: x86 architecture, os: rhel72.
[0025] like figure 1 As shown, backing up a directory of 100G / 1.5 million small files from the client to the storage end includes the following steps:
[0026] Step a1: the storage device shares the logical volume to the client through the network protocol iscsi;
[0027] Step a2: The client formats the received logical volume into the same file system as the source data and mounts it;
[0028] Step a3: The client program retrieves the index node (hereinafter referred to as inode) of the source data directory through stat, directly reads the inode information, generates file metadata to be transmitted, and synchronizes the generated metadata to the logical volume, and generates corres...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention belongs to the field of computer technology, and provides a method for improving the backup efficiency of a large number of small data files, in which the snapshot principle of a block device and the advantages of multi-process parallelism in Linux are combined with the technology of shared memory to realize the backup of a large number of small data files Quick backup. The present invention also provides a method for recovering data files backed up based on the method for backing up a large number of small data files. Under the premise of not compressing the source data and retaining the complete file format and attributes of the source data, the present invention greatly improves the backup speed. Compared with other backup software under the same premise, the backup speed is increased by about 1 / 4. Compared with the backup software for compressed data, The recovery efficiency is greatly improved, and the recovery time after the file is backed up by the method provided by the present invention is almost zero.
Description
technical field [0001] The invention belongs to the technical field of computers, in particular to a method for backing up and restoring a large number of small data files. Background technique [0002] For disk file systems, there are metadata operations regardless of reading and writing. Taking writing data in the EXTx file system as an example, writing data to disk performs a large number of metadata operations, including updating inode directories, directories, inodes, and data block bitmaps. Define the effective data read / write rate Pd=required data / actual disk read / write data, where the actual disk read / write data is the sum of disk metadata and required data. When operating continuous large data files, the operation overhead of metadata can be shared by the huge data operation overhead, but the effective read and write rate of small data files is lower than that of large data files. Data manipulation can seriously affect system performance. [0003] From the above ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F11/14G06F16/11G06F16/13G06F16/178G06F3/06
CPCG06F11/1448G06F16/128G06F16/134G06F16/178G06F3/0665G06F3/067
Inventor 张骏
Owner 南京云信达科技有限公司